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ABSTRACT 


All  too  often,  system  requirements  are  not  funded  appropriately  due  to  failure  to 
incorporate  Human  Systems  Integration  (HSI)  costs  into  the  analysis  of 
alternatives  (AoA)  in  the  acquisition  process.  Failure  to  incorporate  HSI  costs 
results  in  cost  overruns  and  deficiencies  in  performance  that  have  a  significant 
impact  on  life-cycle  costs.  For  a  cost  estimate  to  be  accurate,  all  HSI  domains 
(e.g.,  manpower,  personnel,  training,  human  factors)  must  be  included.  A 
thorough  cost  estimation  process  for  an  AoA  should  identify,  justify,  and  estimate 
all  HSI-specific  activities.  The  current  cost  estimation  processes  are  lacking 
these  HSI-specific  activities. 

This  thesis  describes  the  initial  development  and  evaluation  of  an  HSI  cost 
estimation  process  for  use  during  the  AoA.  A  database  was  created  to  capture  all 
HSI-related  activities  during  each  phase  of  the  acquisition  process.  Forty 
subject-matter  experts  (SMEs)  reviewed  and  provided  feedback  on  the  database 
and  on  a  list  of  HSI  cost  drivers.  They  also  estimated  the  percent  effort  required 
for  each  HSI  activity.  SMEs  concurred  with  80%  of  the  database  and  made  valid 
recommendations  for  improving  the  remaining  20%.  This  initial  framework 
represents  a  first  step  in  providing  a  process  for  estimating  the  cost  of  HSI 
activities. 
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EXECUTIVE  SUMMARY 


All  too  often,  system  requirements  are  not  funded  appropriately  due  to  failure  to 
incorporate  Human  Systems  Integration  (HSI)  costs  into  the  analysis  of 
alternatives  (AoA)  in  the  acquisition  process.  Failure  to  incorporate  HSI  costs 
results  in  cost  overruns  and  deficiencies  in  performance  that  have  a  significant 
impact  on  life-cycle  costs.  For  a  cost  estimate  to  be  accurate,  all  HSI  domains 
(e.g.,  manpower,  personnel,  training,  human  factors)  must  be  included.  A 
thorough  cost  estimation  process  for  an  AoA  should  identify,  justify,  and  estimate 
all  HSI-specific  activities. 

This  thesis  describes  the  initial  development  and  evaluation  of  the  AoA 
HSI  cost  estimation  process.  The  purpose  of  the  HSI  cost  estimation  process  is 
to  support  budget  development  and  justification  over  the  life  cycle  of  the  system. 
The  process  presented  in  this  thesis  complements  the  management  structures 
within  the  Integrated  Defense  Acquisition,  Technology,  and  Logistics  (IDAT&L) 
Life  Cycle  Management.  By  employing  this  process,  HSI  practitioners  and  other 
acquisition  professionals  will  have  access  to  information  to  support  decision 
making  throughout  the  life  cycle  of  the  system  and  to  reduce  total  life  cycle  costs. 
This  process  was  developed  in  conjunction  with  the  HSI  practitioners  from  the 
United  States  Air  Force  and  Navy;  however,  it  can  be  applied  by  all  services  and 
agencies  within  the  Department  of  Defense  and  the  federal  government. 

Two  research  efforts  were  carried  out  to  evaluate  the  utility  of  the  HSI  cost 
estimation  process.  The  first  effort  reviewed  existing  HSI  instructions  and 
guidance,  and  cost  estimation  methodologies.  Information  obtained  from  this 
research  was  used  to  design  a  database  of  specific  HSI  activities  that  require 
cost  estimation  during  to  the  AoA.  The  second  effort  evaluated  the  content  of  the 
database  for  relevance  and  accuracy.  The  database  was  given  to  83  U.S.  Air 
Force  and  Navy  subject-matter  experts  (SMEs)  in  the  fields  of  HSI,  defense 
acquisition  and  cost  estimation;  40  responses  were  received.  SMEs  were 


generally  satisfied  with  the  database.  They  concurred  with  80%  of  the  database 
and  made  valid  recommendations  for  improving  the  remaining  20%.  The  SMEs 
agreed  that,  once  fully  developed,  the  database  would  serve  as  a  basis  for  a 
valuable  HSI-specific  cost  estimation  process  that  could  support  the  HSI 
community  during  the  AoA. 

In  order  to  improve  the  database,  several  SMEs  recommended  identifying 
the  most  important  HSI  activities  in  each  of  the  oversight  documents,  including 
the  following: 

•  Review  Acquisition  Strategy; 

•  Acquisition  Plan; 

•  Source  Selection  Plan; 

•  Request  For  Proposal; 

•  Final  Product  Baseline; 

•  Critical  Safety  List  Items; 

•  Programmatic  Environment,  Safety  and  Occupational  Health 
Evaluation  (PESHE);  and, 

•  TEMP. 

In  addition,  steps  should  be  taken  to  clarify  HSI  activities  in  the  Acquisition 
Management,  Acquisition  Engineering  and  Support/Sustainability  activity  tiers. 
Other  recommendations  included  reorganizing  the  database  in  terms  of  work 
breakdown  structure  cost  elements,  identifying  the  benefits/impacts  and  risk/cost 
relationships  of  HSI  activities. 

Further  development  and  evaluation  of  the  database  is  required  beyond 
what  was  completed  for  this  thesis.  These  refinements  include  comparing  HSI 
cost  methods  and  developing  the  HSI  variables  that  could  be  included  in  a  cost 
model;  determining  the  domain  tradeoffs  that  must  occur  to  achieve  accurate 
cost  estimates;  and,  using  a  system  of  interest  in  a  case  study  to  validate  the 
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database.  The  initial  framework  developed  in  this  thesis  represents  a  first  step  in 
providing  a  process  for  estimating  the  cost  of  HSI  activities  in  the  acquisition 
process. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Department  of  Defense  (DoD)  Instruction  5000.02  has  directed  Program 
Managers  to  take  steps  to  include  HSI  considerations  into  their  planning, 
development,  design  and  execution  of  programs.  This  is  necessary  in  order  to 
minimize  total  ownership  costs  and  also  to  optimize  total  system  performance  as 
early  as  possible  in  the  acquisition  process  (Department  of  Defense,  2008c).  In 
order  to  accomplish  these  tasks,  line  item  costs  from  approved  legacy  system 
Work  Breakdown  Structures  (WBS)  are  treated  as  baselines  for  component 
equipment  (hardware,  software,  etc.)  as  well  as  manpower,  then  incorporated 
into  the  estimates  for  funded  requirements.  A  formal  cost  estimation  process  for 
the  components  is  also  completed.  All  too  often,  requirements  are  not  funded 
appropriately  due  to  failure  to  incorporate  HSI  costs  into  the  Analysis  of 
Alternative  process  for  various  military  system  capabilities.  These  failures 
produce  cost  overruns  and  deficiencies  in  performance  resulting  in  overall 
significant  impact  on  the  life-cycle  cost.  The  development  of  a  standardized  HSI 
cost  estimation  model  is  needed.  Such  an  HSI  cost  estimation  model  would  fulfill 
the  requirements  of  the  system  and  define  the  system  in  terms  of  relevant 
activities  specified  by  cost  estimation  relationships.  The  benefits  and  risks  could 
reduce  the  impact  on  life-cycle  cost.  This  model  would  provide  program 
managers  with  an  enhanced  decision-making  capability. 

To  provide  appropriate  measures,  the  methodologies  need  to  encompass 

size  or  complexity  drivers  and  life  cycle  phase  involvement.  The  methodologies 

must  be  data-driven,  with  activity  effort  qualification  and  quantification.  It  is 

important  to  note  that  the  approaches  listed  in  this  thesis  are  usually  country  or 

service  specific.  Current  methodology  for  HSI  estimates  have  been  performed 

involving  risk  and  benefit  assessment,  activity/cost  factors  and  categorically 

defined  approaches.  An  example  of  past  cost  avoidance  methodology  research 
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includes  a  study  completed  by  the  Canadian  organization  Defence  Technology 
Center  (2006).  This  study  provided  five  arguments  with  supporting  case  studies 
recommending  Human  Factors  Integration  (Canadian-termed  HSI)  inclusion  into 
cost  estimates.  These  estimates  were  included  early  in  the  acquisition  process 
due  to  development  or  operational  risk  impact  on  programs.  The  specific 
arguments  of  the  study  encompassed  reduction  of  cost  (i.e.,  accident  cost, 
redesign  costs),  technology  development  (i.e.,  complexity  induced  human  error), 
identification  and  mitigation  of  operational  risks  (i.e.,  user  involvement  to  ensure 
project  success),  resources  (i.e.,  multiple  benefit  activities)  and  early  involvement 
in  project  development  to  ensure  success. 

Research  is  currently  ongoing  to  specify  and  define  HSI  activities  and  cost 
factors  for  research  efforts  by  civilian  and  government  agencies.  A  study  by  Liu, 
Rhodes  and  Valerdi  (2009)  investigated  including  HSI  into  the  cost  estimates.  Liu 
et  al.  (2009)  provided  a  case  study  of  the  Pratt  &  Whitney  F119  engine,  which 
powers  the  F-22  Raptor  fighter  aircraft.  In  this  case,  the  Air  Force,  in  response  to 
the  need  to  reduce  life  cycle  cost,  created  the  Reliability,  Maintainability  and 
Sustainability  (RM&S)  program  in  1984.  The  RM&S  program  increased 
requirements  for  logistics  and  cost  emphasis  in  the  design  of  the  F-22  engines. 
The  push  allowed  for  a  competition  in  engine  development  by  General  Electric 
(GE)  and  Pratt  &  Whitney.  GE  chose  improved  F120  engine  performance  over 
using  the  RM&S  standards.  Pratt  &  Whitney  surpassed  GE  by  using  integrated 
product  development  teams  (IPT)  with  group  involvement  of  seven  out  of  nine 
domains  of  HSI  (i.e..  Survivability,  Manpower,  Personnel,  Safety,  Training, 
Human  Factors  Engineering  and  Occupational  Health).  The  use  of  the  IPTs 
allowed  Pratt  &  Whitney  to  gain  insight  from  different  disciplines,  as  well  as  from 
the  user  population  that  affected  their  design  decisions.  This  leads  to  an 
increased  investment  of  time  in  the  demonstration  and  internal  efforts.  Ultimately, 
HSI  and  other  related  testing  increased  50%  during  the  Engineering  and 
Manufacturing  Development  (E&MD)  phase.  As  a  result  of  Pratt  and  Whitney’s 
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contribution  of  a  management  plan  and  development  schedule,  the  Air  Force 
awarded  them  the  contract.  Pratt  and  Whitney  truly  demonstrated  an 
understanding  of  RM&S  needs. 

The  Canadian  Defence  Technology  Center  conducted  research  from  2000 
to  2004  on  the  application  of  HSI  during  31  Defense  acquisition  programs 
providing  cost  to  benefit  analysis  results  savings  of  $3.33M.  The  methodology 
that  the  organization  used  to  conduct  the  research  was  divided  into  three 
categories  of  HSI  application  based  on  five  domains:  Human  Factor  Engineering, 
Manpower,  Personnel,  Training,  and  combined  System  Safety  and  Health 
Hazard  Assessment.  The  categories  included  the  immediate  saving  of  time  and 
money  that  occurred  during  work  effort,  extrapolated  savings  (early  HSI 
application  produced  saving  money  over  time),  and  uncalculated  savings 
(decisions  resulting  in  saved  lives  or  increased  operational  effectiveness).  Each 
of  these  three  measures  provided  definitive  terminology  and  a  dictionary  of 
activities  that  when  combined,  provided  savings  across  the  life  cycle  of  any 
program. 

Kopardekar  and  Hewitt’s  (2002)  concept  paper  for  the  Federal  Aviation 
Administration  discussed  four  methods  of  conducting  human  factor  cost 
estimates  involving  committees,  parametric  methods  (for  human  factor  impact  or 
type  of  study),  and  heuristic  approaches.  The  study  recommendations  were 
based  on  percentage  point  system  conversions  for  activities  and  complexity  of 
HSI.  At  the  time  of  completion,  a  recommended  approach  could  not  be  specified, 
with  the  exception  that  the  formation  of  committees  for  consistency  of  process 
estimation  would  be  beneficial. 

HSI  domain  cost  methodologies  are  also  scattered  throughout  such 
disciplines  as  Environmental,  Safety  and  Health  (EER  System  Inc,  1998).  The 
cost  analysis  guides  are  predecessors  of  current  HSI  practices  that  can  be 
updated  for  today’s  terminology  and  acquisition  practices.  Together,  these 
applications  can  be  modified  with  existing  technology  and  methodology 
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development.  Together,  these  practices  could  help  produce  a  general  HSI 
system  cost  model  and  support  the  initiative  to  provide  HSI  costs  early  in  the 
acquisition  of  systems. 

1.  Identification  of  HSI  Domain  Concentration  in  Life  Cycie 
Phases 

Each  of  the  previous  studies  recommended  further  work  to  identify, 
quantify  or  qualify  the  activities  of  HSI  within  the  life  cycle  phases.  These  HSI 
activities  must  be  specified  in  order  to  provide  sufficient  cost  relationships  that 
will  support  cost  estimation  procedures.  Examples  of  HSI-defined  activities  in  life 
cycle  phases  are  provided  by  the  Defense  Acquisition  Integrated  framework 
(Bahnmaier  &  Cochrane,  2009),  Pacific  Science  and  Engineering  Group  /  Space 
and  Naval  Warfare  Systems  (SPAWAR)  (Belk,  Gepp,  Risser,  &  Smillie,  2009), 
and  Lindberg  and  Carr  (2007).  The  Defense  Acquisition  Integrated  Framework 
provides  explicit  definitions  and  procedural  guidance  throughout  the  life  cycle 
phases.  With  this  guidance,  the  Naval  Postgraduate  School  has  applied  HSI 
specific  activities  for  interactive  teaching  of  the  Acquisition  life  cycle  that  can  be 
used  for  thesis  development.  These  activities  occur  during  each  relevant  phase 
in  the  life  cycle.  The  Pacific  Science  and  Engineering  Group  and  SPAWAR  have 
developed  an  integrated  framework  outlining  the  HSI  activities  and  their 
interactions  with  each  domain  for  every  phase  of  the  life  cycle.  Lindberg  and  Carr 
(2009)  present  activities  by  domain  and  phase  that  must  be  considered  when 
training  future  HSI  practitioners  in  Air  Force  Education  and  Training  commands. 
These  activities,  the  result  of  weapons  system  skill  analysis,  are  relevant  to  cost 
estimates  because  task  performance  is  based  on  the  level  of  manpower  effort  (or 
fully  burdened  cost). 

Other  service-centric  examples  of  activity  assessments  are  also  available. 
The  Army’s  MANPRINT  Enterprise-tracking  Analyzer  tool  (META)  provides 
activity-domain  relationships  for  various  programs  by  life  cycle  phase.  In  the  life 
cycle  phase,  manpower  effort  level  and  time  duration  of  HSI  practitioners  are 
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available.  The  Air  Force  HSI  Office  developed  an  integrated  tool  similar  to  the 
DAU  integrated  framework  (Booz  Allen  Hamilton  Inc.,  2009),  one  that  aligns  the 
HSI  activities  with  the  life  cycle  phase  of  DoD. 

All  of  the  processes  listed  above  are  in  various  stages  of  development  and 
are  specific  to  the  armed  service  that  funded  the  creation  of  the  application. 
Currently,  cost  estimation  functions  have  not  been  applied  to  the  frameworks. 
These  frameworks  can  contribute  to  a  standard  HSI  domain  activity-driven 
framework  that  is  useful  for  work  breakdown  structure  (WBS)  element  or  Cost 
Element  Structure  (CES)  development  for  cost  estimation.  Furthermore,  each  of 
these  frameworks  has  yet  to  be  compiled  to  form  a  standard  process  that  has 
been  adopted  by  DoD  to  pursue  cost  estimates. 

2.  Problem  Statement 

Given  the  aforementioned  research  and  prototype  applications  currently 
under  development,  a  standard  cost  estimation  methodology  for  HSI  -one  that 
can  easily  be  manipulated  to  fit  the  needs  of  any  system  still  has  not  been 
developed. 

The  USAF’s  711th  Human  Performance  Wing  (HPW)  has  taken  on  the 
task  from  the  Cffice  of  Aerospace  Studies  of  creating  cost  estimation  tools  for 
HSI.  The  711th  HPW  began  the  task  by  considering  HSI  cost  estimation 
practices  during  the  AoA  portion  of  the  Defense  acquisition  process.  For  a  cost 
estimate  to  be  credible,  a  necessary  condition  is  that  it  be  comprehensive;  i.e.,  all 
of  the  HSI  domains  (manpower,  personnel,  training,  human  factors,  etc.)  must  be 
included.  A  formal  process  of  cost  estimation  for  an  AoA  should  identify,  justify 
and  estimate  all  HSI-specific  activities. 

This  cost  estimation  process  for  HSI  activities  should  be  coordinated  with 
current  USAF  process  stakeholders  for  Manpower,  Personnel,  Training, 
Environmental,  Safety  and  Occupational  Health.  Furthermore,  the  estimation 
process  should  integrate  all  domains  of  HSI  within  services  to  the  greatest  extent 
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possible,  starting  with  the  Capabilities-Based  Assessment  and  continuing 
through  Full  Operational  Capability.  Prior  to  undertaking  the  development  of  a 
cost  estimation  methodology,  an  HSI  Concept  Model  is  needed  to  establish 
domain  relationships  within  each  of  the  four  life-cycle  phases  (see  Figure  1).  This 
model  correctly  identifies  those  activities  used  by  HSI  practitioners.  If  a  viable 
cost  estimation  methodology  is  developed,  current  Air  Force  programs,  such  as 
the  unmanned  aerial  system  (LIAS)  will  benefit  from  results  gathered  during  this 
thesis. 


Life  Cycle  Cost  vs.  Time 


Figure  1.  Life  Cycle  Cost  (From  Nussbaum,  2010) 


3.  Analysis  of  Alternatives  Process 

Chairman  of  the  Joint  Chief  of  Staff  Instruction  (CJCSI)  31 70.01  G,  (2009) 
documents  the  general  process  for  establishing  life  cycle  costs  during  the  AoA 
portion  of  the  acquisition  process.  Capability  gaps  assessed  in  the  JCIDS 
process  allow  for  three  courses  of  action: 

•  Accept  the  operational  risk  and  perform  no  action. 
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Seek  doctrine  change  with  a  non-materiel  approach. 
Provide  a  materiel  solution. 


Once  the  Initial  Capabilities  Document  (ICD)  documents  the  requirements 
for  a  materiel  solution,  the  Milestone  Decision  Authority  (MDA)  determines  the 
scope  of  the  subsequent  AoA  production.  Upon  approval  at  the  Milestone 
Development  Decision  (MDD),  the  AoA  study  is  initiated  and  will  guide  the 
Materiel  Solution  Analysis  phase.  Further  updates  to  the  AoA  occur  at  the 
Technology  Development  phase  and  at  Milestone  B  as  seen  in  Figure  2 
(Defense  Acquisition  University,  2010.  p.  95). 


Analytic  Support  to  the 
Defense  Acquisition  Management  System 


CBA 

ICD 

MDD 

AoA 

CDD 


Capabilifties-Based  Assessment 
Initial  Oapabiities  Document 
Materiel  Development  Decision 
Analysis  of  Afternatives 
Capability  Development  Document 


Figure  2.  Analytic  Support  to  the  Defense  Acquisition  Management  System 

(From  Defense  Acquisition  University,  2010) 


B.  OBJECTIVES 

•  This  thesis  will  give  the  USAF’s  711th  FIPW  a  qualitative  and 
quantitative  view  of  FISI-specific  cost  criteria  to  conduct  tradeoff 
considerations  when  making  complex  decisions  within  AoA. 

•  The  thesis  will  benefit  the  AF  HPW  by  developing  a  cost  estimation 
method  or  process  for  the  AoA.  This  method  is  essential  for  current 
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systems  to  support  budget  development  and  justification  over  the 
life  cycle  of  system. 

•  By  providing  this  method  or  process,  HSI  practitioners  and  other 
acquisition  professionals  will  have  improved  information  that  should 
lead  to  better  decisions.  This  process  or  method  may  also  be  useful 
for  other  services  and  agencies  within  the  Department  of  Defense 
and  the  Federal  government. 

•  The  method  or  process  will  allow  HSI  practitioners  and  other 
professionals  the  capability  to  assess  and  mitigate  potential  risks 
and  uncertainties  that  will  influence  the  future  development  of 
systems. 

C.  RESEARCH  QUESTIONS 

•  Where  do  HSI  activities  fall  within  the  life  cycle  phases  of  DoD 
acquisition? 

•  What  HSI  cost  estimation  methodologies  are  currently  used  and 
why? 

•  What  time  durations  or  level  of  effort  do  HSI  practitioners  consider 
when  performing  HSI  activities? 

•  What  are  the  metric  definitions  used  to  determine  cost  factors  of 
HSI  activities? 

•  What  levels  of  complexity  must  be  considered  when  completing 
HSI  cost  estimates? 

•  What  HSI  costs  are  associated  with  AoAs? 

•  What  HSI  domain  tradeoffs  must  occur  to  achieve  comprehensive 
cost  estimates  with  AoAs? 
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D.  HSI  CONSIDERATIONS 

Manpower,  Personnel,  and  Training  are  the  domains  most  often 
considered  when  estimating  cost.  However,  the  domains  of  Human  Factors 
Engineering,  Environment,  Safety,  Occupational  Health,  Survivability,  and 
Habitability  will  be  also  be  addressed  because  they  are  also  involved  in  tradeoffs 
and  cost/  benefit  analysis.  If  more  cost  factors  are  developed  for  AoAs,  it  should 
lead  to  the  acquisition  of  systems  that  meet  the  needs  of  the  Warfighter  within 
the  cost  constraints  imposed  by  the  Federal  budget. 

The  DoD  Instruction  5000.02  Enclosure  (8)  and  the  DoD  HSI  Management 
Plan  Enclosure  (1)  describe  the  variables  of  HSI  that  should  be  included  by  the 
program  manager  on  various  programs.  To  the  extent  possible,  these  variables 
should  be  integrated  and  receive  tradeoff  consideration  in  terms  of  the  cost  of  the 
system  or  materiel  solution  used.  The  Air  Force  HSI  Plan  states  that  the  activities 
of  HSI  must  be  defined  and  included  in  total  life  cycle  cost.  HSI  activities  should 
also  be  traceable  and  measurable  (i.e.,  level  of  effort).  These  domain  variables 
are  subject  to  available  data,  and  may  not  be  inclusive: 

Manpower.  In  advance  of  contracting  for  operational  support 
services,  the  PM  shall  work  with  the  manpower  community  to 
determine  the  most  efficient  and  cost-effective  mix  of  DoD 
manpower  and  contract  support.  The  mix  of  military,  DoD  civilian, 
and  contract  support  necessary  to  operate,  maintain,  and  support 
(to  include  providing  training)  the  system  shall  be  determined  based 
on  the  Manpower  Mix  Criteria  and  reported  in  the  Manpower 
Estimate.  Economic  analyses  used  to  support  workforce  mix 
decisions  shall  use  costing  tools  that  account  for  fully  loaded  costs- 
i.e.,  all  variable  and  fixed  costs,  compensation  and  non¬ 
compensation  costs,  current  and  deferred  benefits,  cash  and  in- 
kind  benefits.  (Department  of  Defense,  2008c,  p.  60;  Department  of 
Defense,  2009b,  p.  4) 

Personnel:  To  the  extent  possible,  systems  shall  not  require  special 
cognitive,  physical,  or  sensory  skills  beyond  that  found  in  the 
specified  user  population.  For  those  programs  that  have  skill 
requirements  that  exceed  the  knowledge,  skills,  and  abilities  of 
current  military  occupational  specialties,  or  that  require  additional 
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skill  indicators  or  hard-to-fill  military  occupational  specialties,  the 
PM  shall  consult  with  personnel  communities  to  identify  readiness, 
personnel  tempo,  and  funding  issues  that  impact  program 
execution.  The  acquisition  strategy  and  Life-Cycle  Sustainment 
Plan  should  address  modifications  to  the  knowledge,  skills,  and 
abilities  of  military  occupational  specialties  for  system  operators, 
maintainors,  or  support  personnel  if  the  modifications  have  cost  or 
schedule  issues  that  could  adversely  impact  program  execution. 
The  program  manager  should  consider  personnel  factors  such  as 
availability,  recruitment,  skill  identifiers,  promotion,  and  assignment. 
The  program  manager  should  consider  the  impact  on  recruiting, 
retention,  promotions,  and  career  progression  when  establishing 
program  costs,  and  should  assess  these  factors  during  trade-off 
analyses.  (Department  of  Defense,  2008c,  p.  60;  Defense 
Acquisition  University,  2010) 

Training:  The  PM  shall  work  with  the  training  community  to  develop 
options  for  individual,  collective,  and  joint  training  for  operators, 
maintainors  and  support  personnel,  and,  where  appropriate,  base 
training  decisions  on  training  effectiveness  evaluations.  The  PM 
shall  address  the  major  elements  of  training,  and  place  special 
emphasis  on  options  that  enhance  user  capabilities,  maintain  skill 
proficiencies,  and  reduce  individual  and  collective  training  costs. 
The  PM  shall  develop  training  system  plans  to  maximize  the  use  of 
new  learning  techniques,  simulation  technology,  embedded  training 
and  distributed  learning  (DoD  Instruction  1322.26  (Reference 
(be))),  and  instrumentation  systems  that  provide  “anytime, 
anyplace”  training  and  reduce  the  demand  on  the  training 
establishment.  Where  possible,  the  PM  shall  maximize  the  use  of 
simulation-supported  embedded  training,  and  the  training  systems 
shall  fully  support  and  mirror  the  interoperability  of  the  operational 
system.  Identify  training  requirements  and  cost  to  maintain/refresh 
knowledge,  skills,  and  abilities,  and  training  or  develop  personnel 
attributes  and  training  not  currently  available  to  the  Navy.  Conduct 
affordability  assessments  that  include  estimates  of  the  TOC  of  the 
training  infrastructure,  manpower,  and  the  training  associated  with 
each  approach.  (DoD  Directive  1322.18  (Reference  (bf); 
Department  of  Defense,  2008c,  p.  61;  Department  of  Defense, 
2009b,  p.  4) 

Survivability:  For  systems  with  missions  that  might  require 
exposure  to  combat  threats,  the  PM  shall  address  personnel 
survivability  issues  including  protection  against  fratricide,  detection, 
and  instantaneous,  cumulative,  and  residual  nuclear,  biological. 
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and  chemical  effects;  personnel  survivability  against  asymmetric 
threats;  the  integrity  of  the  crew  compartment;  and  provisions  for 
rapid  egress  when  the  system  is  severely  damaged  or  destroyed. 
The  PM  shall  address  special  equipment  or  gear  needed  to  sustain 
crew  operations  in  the  operational  environment,  including  the 
suitability  of  equipment  intended  to  enhance  personnel  survivability 
against  asymmetric  threats.  (Department  of  Defense,  2008c,  p.  61) 

Environmental,  Safety  and  Occupational  Health:  The  PM  shall 
ensure  that  appropriate  HSI  and  ESOH  efforts  are  integrated 
across  disciplines  and  into  systems  engineering  to  determine 
system  design  characteristics  that  can  minimize  the  risks  of  acute 
or  chronic  illness,  disability,  or  death  or  injury  to  operators  and 
maintainors;  and  enhance  job  performance  and  productivity  of  the 
personnel  who  operate,  maintain,  or  support  the  system. 
(Department  of  Defense,  2008c,  p.  61)  Costs  can  be  associated 
with  functions  within  legacy  systems  increasing  accident  rates  or 
hazardous  environments  that  predispose  long-term  medical  related 
costs.  Costs  associated  with  concept  development  allowing  for 
mitigation  of  safety  and  health  issues  (e.g.  warning  systems, 
enhanced  safety  procedures)  will  also  be  relevant. 

Habitability:  The  PM  shall  work  with  habitability  representatives  to 
establish  requirements  for  the  physical  environment  (e.g.,  adequate 
space  and  temperature  control)  and,  if  appropriate,  requirements 
for  personnel  services  (e.g.,  medical  and  mess)  and  living 
conditions  (e.g.,  berthing  and  personal  hygiene)  for  conditions  that 
have  a  direct  impact  on  meeting  or  sustaining  system  performance 
or  that  have  such  an  adverse  impact  on  quality  of  life  and  morale 
that  recruitment  or  retention  is  degraded.  (Department  of  Defense, 
2008c,  p.  60) 

Human  Factors  Engineering:  Where  practicable  and  cost  effective, 
system  designs  shall  minimize  or  eliminate  system  characteristics 
that  require  excessive  cognitive,  physical,  or  sensory  skills;  entail 
extensive  training  or  workload-intensive  tasks;  result  in  mission- 
critical  errors;  or  produce  safety  or  health  hazards.  Identify  MPT 
TOC  savings  achieved  through  analysis  of  design  human  factors, 
workload,  and  associated  KSAs,  resources,  and  tools  for  all  tasks 
allocated  to  humans  required  for  the  efficient  operation, 
employment,  and  support  of  the  system.  (Department  of  Defense, 
2008c,  p.  60;  Department  of  Defense,  2009b,  p.  3) 
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E.  THESIS  ORGANIZATION 

Chapters  pertaining  to  the  development  of  the  final  product  will  organize 
this  thesis.  Chapter  II  reviews  the  literature  that  was  used  to  develop  and 
establish  the  cost  estimation  process.  Chapter  II  will  be  broken  into  six  parts. 

Part  One  reviews  the  cost  estimate  decision  support  for  thesis 
development.  The  section  includes  the  statutory  and  regulatory  basis  for  cost 
estimates,  Department  of  Defense  and  life  cycle  management  processes,  Joint 
Capabilities  Integration  Development  System  (JCIDS)  processes  and  HSI 
relevance  within  the  acquisition  life  cycle.  Part  Two  reviews  cost  estimation 
techniques,  methods  and  processes  to  support  thesis  development.  The  section 
discusses  the  scope,  estimate  type  purposes,  rules  and  assumptions,  and 
estimation  techniques  for  this  thesis.  Further  discussion  will  include  modeling  and 
identification  of  cost  drivers  as  well  as  cost  estimation  methodologies  to  be  in  HSI 
effort  determination.  Part  Three  discusses  the  HSI  Concept  Model  that  will  be 
used  to  develop  our  HSI  cost  estimation  process.  Part  Four  will  discuss  relevant 
cost  estimation  methods  that  are  currently  used  for  HSI  cost  estimation.  Part  Five 
will  cover  the  system  of  interest  for  practical  application,  validation  and 
verification  of  potential  HSI  cost  estimation  processes  derived  during  thesis 
development. 

Chapter  III  will  discuss  the  method  or  processes  to  be  used  in  the  analysis 
of  data.  Chapter  IV  will  cover  the  results  of  the  analysis  of  data.  Chapter  V  will 
discuss  relevant  lessons  learned.  Finally,  Chapter  VI  will  discuss  the  way 
forward. 
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II.  LITERATURE  REVIEW 


The  process  of  cost  estimation  goes  beyond  merely  performing 
calculations  for  AoA  or  for  congressional  reports.  Cost  estimation  is  an 
amalgamation  of  support  systems  that  helps  ensure  the  validity,  reproducibility 
and  affordability  of  any  defense  acquisition  program.  As  stated  earlier,  the  AoA 
starts  with  Warfighter  requirement  determination.  The  requirements  must  meet 
the  policy  and  strategy  guidance,  after  which  they  must  receive  approval  from 
Program,  Budget  and  Execution  processes  before  system  capabilities  can  be 
produced.  The  focus  of  this  chapter  is  to  discuss  these  processes  as  well  as  the 
HSI  activities  that  should  be  used  for  cost  estimation.  This  chapter  discusses  the 
following: 

•  Cost  estimation  decision  support  system, 

•  Cost  estimation  methods, 

•  HSI  concept  models, 

•  Current  HSI  cost  methods. 

A.  COST  ESTIMATE  DECISION  SUPPORT  SYSTEM 

This  section  discusses  the  decision  support  system  and  HSI  activity 
development  in  the  Life  Cycle  Management  system.  Parts  one  through  three 
discuss  the  decision  support  system  including  the  Joint  Capabilities  Integration 
and  Development  System  (JCIDS);  Acquisition  Management  and  the  DoD 
Planning,  and  Programming,  Budget  and  Execution  (PPBE)  process.  Part  four 
discuss  HSI  processes  in  defense  acquisition  that  require  funding  and/or 
manpower  support.  These  processes  are  used  in  the  development  of  a  cost 
estimation  process  for  HSI  activities. 
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1.  statutory  and  Regulatory  Requirements  for  Cost  Estimates 

The  requirement  for  cost  estimation  is  found  in  the  United  States  Code 
(USC)  Title  10,  Sections  2430  through  and  2435.  The  USC  states  that  a  full  life 
cycle  cost  must  be  reported  for  each  major  defense  acquisition  program.  The 
Secretary  of  Defense  cannot  approve  the  life  cycle  cost  unless  an  independent 
cost  estimate  (ICE)  of  the  full  life  cycle  of  the  program  has  been  completed.  The 
ICE  must  be  conducted  by  entities  other  than  the  organization  directly 
responsible  for  the  program,  regardless  of  fund  source  or  management  control 
(USC,  2006,  pp.  1194-1199).  USC  (2006)  section  2433a  states  that  an 
Acquisition  Category  (ACAT)  I  program  must  have  approval  of  the  Director  of 
Cost  Assessment  and  Program  Evaluation  (DCAPE)  in  order  to  proceed  to  the 
Technology  Development  (TD),  E&MD,  and  Production  and  Deployment  (P&D) 
stages. 

Key  regulatory  costing  events  or  items  are  mandatory  inputs  to  the 
milestone  decisions  that  are  made  at  the  end  of  the  TD,  E&MD,  and  P&D  stages. 
These  events  are  as  follows:  Cost  Analysis  Requirements  Description  (CARD), 
Life  Cycle  Cost  Estimates  (LCCE)  and.  Service  Cost  Position  (SCP)  / 
Independent  Cost  Estimate  (ICE)  (See  Figure  3). 

The  CARD  provides  a  common  technical  description  and  a  system 
baseline,  as  well  as  technical  descriptions  for  all  organizations  providing  cost 
projections  (Nussbaum,  2010).  The  CARD  is  reviewed  at  each  milestone.  The 
next  section  discusses  the  LCCE  and  ICE  cost  events. 
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Figure  3.  Key  Regulatory  Costing  Events  leading  to  a  DAB  Milestone 

Decision  (From  Nussbaum,  2010) 


2.  Department  of  Defense/Life  Cycle  Management  Process 

DoDI  5000.02  and  DoD  Directive  5000.04  provide  further  clarification  for 
development  of  LCCE,  ICE  requirements  for  ACAT  programs.  Details  for 
establishment  of  the  Office  of  Cost  Assessment  (OCA)  and  Director  of  Cost 
Assessment  and  Program  Evaluation  (DCAPE)  are  also  provided. 

When  a  program  is  initiated,  the  service  program  office  must  produce  a 
Program  Office  Estimate  (POE).  If  ACAT  ID  programs  involve  significant  risk  cr 
are  considered  highly  visible,  a  Component  Cost  Analysis  (CCA)  will  also  be 
prepared.  The  Component  Acquisition  Executive  requests  the  CCA.  Specifically, 
DoDI  5000.02  states: 

For  ACAT  ID  or  ACAT  IC  programs,  the  independent  cost  estimate 
is  required  prior  to  certification  at  Milestone  A  (see  10  U.S.C. 
2366a),  Milestone  B  (see  10  U.S.C.  2366b),  and  before  any 
decision  to  enter  into  low-rate  initial  production  or  full-rate 
production.  Independent  cost  estimates  are  also  required  in 
advance  of  certification  following  critical  ccst  grcwth  in  majcr 
defense  programs  (pursuant  to  10  U.S.C.  2433a).  In  addition,  the 
Office  of  Cost  Assessment  for  an  ACAT  ID  program  may  conduct 
an  independent  cost  estimate  at  any  other  time  as  directed  by 
either  the  DCAPE  or  the  USD  (AT&L).  (Defense  Acquisition 
University,  2010;  Department  of  Defense,  2008c) 
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Although  an  ICE  is  not  required  for  ACAT  lA  programs,  the  DoD  5000 
series  requires  a  CCA  at  Milestone  B  and  whenever  an  Economic  Analysis  is 
requested  by  the  Milestone  Decision  Authority  (MDA).  The  CCA  and  Economic 
Analysis  estimates  are  completed  by  the  OSD  Office  of  Cost  Assessment  staff  or 
sponsoring  DoD  components.  Furthermore,  milestone  and  decision  reviews 
require  a  POE.  However,  life  cycle  benefits  and  costs,  which  are  subject  to  the 
program’s  integrated  product  team  approval,  are  required  for  the  ACAT  lA  POE 
(Department  of  Defense,  2008c). 

DoDI  5000.02  indicates  that  all  DoD  components  (Military  Departments 
and  Defense  Agencies)  prepare  LCCEs  in  support  of  their  acquisition  programs. 
“A  LCCE  attempts  to  identify  all  the  costs  of  an  acquisition  program,  from  its 
initiation  through  disposal  of  the  resulting  system  at  the  end  of  its  useful  life  and 
to  properly  phase,  or  spread,  the  costs  for  inclusion  in  budget  submission 
documents”  (Department  of  Defense,  2008c;  Defense  Acquisition  University, 
2010).  To  account  for  affordability,  the  LCCEs  are  used  during  the  review 
process  to  ensure  system  affordability  and  to  provide  budgetary  considerations 
for  Congress.  Integrated  Product  Teams  (IPTs)  and  an  Integrated  Product  and 
Process  Development  (IPPD)  are  typically  employed  to  ensure  LCCEs  are 
completed  accurately.  LCCEs  are  used  in  the  Planning,  Program,  Budgeting  and 
Execution  process  (PBBE)  which,  in  turn,  is  used  for  the  funding  requests  that 
are  sent  to  Congress. 

The  last  regulatory  action  to  discuss  is  the  establishment  of  the  Office  of 
Cost  Assessment  (OCA)  and  the  DCAPE.  DoD  Directive  5000.04  states  that  “the 
OSD  OCA  provides  independent  analysis  and  advice  to  DoD  officials  on  matters 
of  cost  estimation  and  cost  analysis  for  weapons  acquisition  programs,  including 
matters  of  program  life  cycle  cost.  The  DCAPE  provides  policies  and  procedures 
for  the  conduct  of  all  DoD  cost  estimates.  The  DCAPE  also  issues  guidance 
relating  to  the  full  consideration  of  life-cycle  management  and  sustainability 
costs.  In  addition,  the  DCAPE  reviews  DoD  Component  cost  estimates  and  cost 
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analyses  conducted  in  connection  with  Major  Defense  Acquisition  Programs 
(MDAP)  and  Major  Automated  Information  Systems  (MAIS)”  (Department  of 
Defense,  2008b). 

3.  Joint  Capabilities  Integration  Development  System  (JCIDS) 

This  section  discusses  the  principles  that  underlie  JCIDS.  Implemented  in 
2002,  the  JCIDS  process  evolved  because  the  requirements  process  was  flawed. 
The  three  underlying  principles  of  the  JCIDS  process  include:  (a)  Determining  the 
needs  from  a  joint  perspective,  (b)  describing  those  needs  in  terms  of 
capabilities,  and  (c)  overseeing  the  DoD  functional  portfolio  by  a  flag  officer. 
These  principles  lead  to  the  identification  of  the  capabilities  needed  rather  than  to 
a  specific  materiel  solution  (i.e.,  system  function  need  versus  system 
requirement).  A  joint  perspective  allows  for  system  development  instead  of 
parallel  development,  which  may  produce  duplicate  costs.  Finally,  the  process, 
overseen  by  a  flag  officer,  allows  the  scope  of  the  solution  to  meet  joint  service 
needs  rather  than  narrow  requirements  suiting  one  function. 

The  JCIDS  process  was  revised  in  2009  to  include  a  Capability  Based 
Assessment  (CBA).  The  revision  removed  the  processes  of  Functional  Area 
Analysis,  Functional  Needs  Analysis  and  the  Functional  Solutions  Analysis.  As 
the  entry  point  into  the  acquisition  process,  the  CBA  identifies  gaps  and  provides 
recommendations  for  closing  the  gap.  As  stated  in  Chapter  I,  this  process 
eventually  leads  to  a  DOTMLPF  Change  Recommendation  (DCR)  and/or  an  AoA 
for  a  materiel  and/or  non-materiel  solution. 

It  is  important  to  consider  HSI  early  in  order  to  mitigate  total  cost.  HSI 
consideration  begins  during  the  CBA  when  HSI  subject-matter  experts  provide 
input  for  activities  that  will  eventually  be  inserted  into  the  Initial  Capabilities 
Document  (ICD)  and  Capabilities  Development  Document  (CDD)  (See  Figure  4, 
Simpkiss,  2009).  A  CBA  should  consider  the  full  spectrum  of  solutions,  including 
doctrine,  organization,  training,  materiel,  leadership  and  education,  personnel. 
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and  facilities  (DOTMLPF).  If  the  ICD  recommends  a  materiel  solution,  an  AoA 
study  plan  is  created.  HSI  practitioners  can  contribute  to  any  of  these 
alternatives. 


Figure  4.  Human  System  Integration  in  Requirements  (From  Simpkiss,  2009) 

Once  the  CBA  has  been  completed,  an  ICD  is  written.  Warfighter  needs 
are  identified  in  the  JCIDS  process  and  are  developed  into  operational  terms  for 
the  ICD.  These  operational  terms  are  then  molded  into  specific  requirements  in 
the  CDD.  The  Joint  Requirements  Oversight  Committee  (JROC)  and  the  Defense 
Acquisition  Board  (DAB)  monitor  and  review  the  JCIDS  activities  throughout  the 
acquisition  life  cycle. 

The  past  three  sections  covered  statutory  and  regulatory  requirements  for 
cost  estimation,  the  Life  Cycle  Management  Process  and  the  Joint  Capabilities 
Integration  and  Development  System.  The  next  section  discusses  the  HSI 
activities  that  should  be  considered  part  of  the  cost  estimation  activities  that 
occur  in  the  acquisition  process. 
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4. 


Human  System  Integration  in  the  Acquisition  Life  Cycie 


The  Air  Force  HSI  Requirements  Pocket  Guide  (2009)  and  current  service 
policies  provide  an  excellent  overview  of  the  HSI  activities  in  each  phase  of  the 
acquisition  life  cycle.  These  documents  will  be  used  to  explain  the  relevant  HSI 
activities  in  each  phase  and  their  associated  cost  metrics.  The  Air  Force  Pocket 
Guide  (2009)  provides  a  detailed  description  of  the  HSI  effort  within  the 
Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life  Cycle 
Management  System  framework.  These  policies  were  chosen  because  of  the 
abundance  of  use  in  the  HSI  community.  Other  policies  exist  such  as  MANPRINT 
(AR  602-2).  The  MANPRINT  Handbook  was  review  but  did  not  add  any 
additional  insights  to  the  present  discussion. 

Each  military  service  has  its  own  policies  that  mandate  how  HSI  should  be 
accomplished  in  the  acquisition  process.  Four  current  service  policies  that  are 
particularly  informative  are  the  (a)  OPNAVINST  5310.23,  Navy  Personnel  Human 
System  Integration  (NAVPRINT),  (b)  Air  Force  Instruction  (AFI)  10-601, 
Capabilities  Based  Requirements  Development,  (c)  AFI  63-1201,  Life  Cycle 
Systems  Engineering  and  (d)  AFI  63-101,  Operations  of  Capability  Based 
Acquisition  Systems. 

OPNAVINST  5310.23D  Enclosure  (1)  specifies  that: 

The  CBA  must  account  for  current  and  projected  manpower  and 
personnel  characteristics.  CBA  analysis  must  identify  operational  or 
environmental  conditions  that  may  impact  performance  of  the 
operators,  maintainors  and  support  personnel  for  future  platforms 
and  systems. 

The  AoA  provides  the  foundation  for  KPPs,  key  system  attributes 
(KSA),  and  other  attributes  in  CDDs  and  CPDs;  therefore  it  is  vitally 
important  that  HSI  considerations,  where  applicable,  be  evaluated 
during  the  AoA.  This  allows  HSI  considerations  (e.g.,  manpower 
and  systems  training)  and  their  associated  costs,  to  be  addressed 
when  selecting  the  preferred  alternative.  (Department  of  Defense, 

2009e) 
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The  Department  of  the  Air  Force  Instruction,  AFI  63-1201  emphasizes  the 
incorporation  of  MSI  into  the  System  Engineering  Plan  (SEP)  with  specific 
guidance  for  Environment,  Safety,  Occupational  Health  and  Habitability 
requirements.  Specifically  addressed  are  the  Maintenance  and  Support 
personnel,  who  are  the  backbone  of  any  operation.  The  document  states:  “It 
must  be  addressed  throughout  the  life  cycle,  and  must  be  consistently  integrated 
into  SE  implementation  to  balance  total  system  performance  (hardware, 
software,  and  human),  assurance  of  Operational  Safety,  Suitability,  and 
Effectiveness  (OSS&E),  survivability,  and  affordability”  (Department  of  Defense, 
2007a). 

Air  Force  Instruction  63-101  (2009)  states  that  “the  HSI  office  will  oversee 
and  advocate  an  HSI  focus  in  activities  regarding  systems  integration,  systems 
engineering,  total  system  performance  and  total  operating  costs”  (pg.  41).  This 
instruction  highlights  the  system  engineering  process  by  stating:  “ESOH 
integration  strategy  must  define  HSI  roles  and  activities  for  those  domains” 
(Department  of  Defense,  2009a).  Emphasizing  trade-offs  between  HSI  domains, 
the  ESOH  integration  strategy  incorporates  survivability  of  the  system.  In 
accordance  with  these  instructions,  the  survivability  of  the  system  should  be 
considered  during  trade-off  analysis  when  considering  affordability,  schedule, 
and  performance  capabilities. 

Clearly,  an  HSI  focus  is  mandatory  for  Navy  and  Air  Force  acquisition 
programs.  The  next  step  is  to  examine  the  HSI  activities  that  occur  during  each 
phase  of  the  acquisition  process  and  to  consider  the  cost  associated  with  these 
activities. 


a.  Materiel  Solution  Analysis  (MSA)  Phase 

The  focus  of  the  MSA  phase  is  ‘completing  Analysis  of  Alternatives 
to  assess  potential  materiel  solutions  to  capability  need,  identifying 
key  technologies  and  estimate  life  cycle  costs’.  This  phase 
‘considers  commercial-off-the-shelf  solutions  from  both  large  and 
small  business  and  identifies  materiel  solution  to  capability  need’. 
(Defense  Acquisition  University  Online,  2010) 
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The  overall  goal  of  the  MSA  phase  is  to  identify  an  affordable, 
operationally  effective  materiel  solution.  During  this  phase,  inputs  are  provided 
for  MDA  cost  estimates,  AoA  study  plan/analysis,  study  contracts.  Technology 
Development  Strategy  (TDS),  Request  for  Proposals  (RFP)  and  draft  CDD 
(Department  of  Defense,  2008c).  The  cost  estimates  given  during  this  phase 
allow  the  MDA  to  determine  adequate  funding  of  the  Technology  Development 
Phase  for  the  preferred  materiel  solution.  In  this  section,  relevant  HSI  activities 
are  described  from  current  resources  including:  (a)  SECNAV-5000.2  Navy 
Acquisition  and  Capabilities  Guidebook,  (b)  Defense  Acquisition  Guidebook,  (c) 
DoD  5000  series  instruction,  (d)  2010  Weapon  System  Reform  Act  (WSARA) 
and  the  (e)  Human  Systems  Engineering  Manual. 


SECNAVM-5000.2  states  the  importance  of  HSI  activities  within  the 
Technology  Development  Strategy  (TDS)  (See  Figure  5).  The  TDS  includes:  (a) 
Determining  issues  related  to  physical/mental  characteristics,  (b)  describing  the 
target  audience  description  (TAD),  and  (c)  providing  performance  parameters  for 
operators  and  maintainers.  When  determining  affordability,  feasibility,  and  risk, 
HSI  practitioners  must  also  consider  training,  manpower,  job  specialization 
(Personnel  Rates/MOS),  and  technology  insertion  (DoD  [2008d];  Enclosure  7). 


IDAT&L  Life  Cycle  Mgmt  System 


Technology  Development  Strstegy 

•  Acquisition  Approach 

•  Resource  Management 

*  Source  and  Related  Documents 

•  Program  Security  Considerations 

*  Capability  Needs 

•  Test  Planning 

•  Top-Level  Integrated  Schedule 

'  Data  Management  &  Technical  Data 

•  Program  Interdependency  & 

Rights 

Interoperability  Summary 

•  Life-Cycle  Sustainment  Planning 

•  International  Cooperation 

•  Life-Cycle  Signature  Support  Plan 

•  Risk  &  Risk  Management 

-  Technology  Maturation 

•  CBRN  Survivability 

-  Industrial  Capability  &  Manufacturing 
Capabilities 

'  Business  Strategy 

Figure  5.  Technology  Development  Strategy  Contents  (From  Defense 

Acquisition  University,  2010) 
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The  second  resource  is  the  Defense  Acquisition  Guidebook  (DAG), 
Chapter  3.  The  DAG  describes  the  AoA  process  and  begins  with  a  detailed 
structure  of  the  AoA  Study  Plan.  Included  in  the  plan  are  analysis  for  scenario 
and  threat  development  (bound  by  physical  and  environmental  constraints). 
Capability  ranges  are  also  used  to  determine  testing  scenarios,  while  operation 
effectiveness  measures  are  used  to  associate  metrics  for  military  worth  with  the 
alternatives  (i.e.,  mission  outcome,  system  performance,  identification  of 
simulation  models). 


Life-cycle  cost  analysis  will  be  a  part  of  the  study  plan.  The  life 
cycle  cost  estimate  is  combined  with  the  operational  effectiveness  analysis  for  a 
full  cost-effectiveness  comparison  portrayal  (see  Figure  6).  The  final  segment  of 
the  AoA  Study  Plan  includes  the  organization  and  management  structure  for 
conducting  the  AoA,  as  well  as  any  oversight  and  guidance  needed.  With  the 
AoA  Study  Plan  in  place,  the  AoA  can  proceed  as  described  in  the  next 


paragraph. 
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Figure  6.  Sample  Scatter  Plot  of  Effectiveness  versus  Cost  comparison 
(From  Defense  Acquisition  University,  2010) 
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The  HSI  activities  for  the  AoA  process  are  distributed  throughout 
DoD  5000  series  instructions.  Possible  AoA  materiel  solutions  are  based  on  the 
ability  for  system  alternatives  to  meet/fulfill  concepts  of  operation,  measures  of 
effectiveness,  cost,  schedule,  and  risk  criteria  for  technologies.  Commercial-off- 
the-Shelf  (COTS)  solutions  are  used  if  possible.  DoDI  5000.02  directs  that  “the 
AoA  shall  assess  the  critical  technology  elements  associated  with  each  proposed 
materiel  solution,  including  technology  maturity,  integration  risk,  manufacturing 
feasibility,  and,  where  necessary,  technology  maturation,  and  demonstration 
needs”  (Department  of  Defense,  2008c). 

The  2010  Weapons  System  Acquisition  Reform  Act  (WSARA) 
explicitly  amends  DoDI  5000.02  and  the  Defense  Federal  Acquisition  Regulation 
Supplement  to  require  competition  for  prototype  development.  Fox  (2010)  stated 
that  the  2010  WSARA  will  help: 

•  Identify  cost  risk  earlier, 

•  Baseline  programs  with  realistic  cost  and  schedule  estimates, 

•  Ensure  rigors  of  AoA, 

•  Ensure  Departments  fully  fund  development  and  procurement  cost 
of  programs,  and 

•  Lead  Departments  in  improving  cost  assessment  and  analysis. 

The  implementation  of  the  2010  WSARA  affects  performance  assessments  of  an 
AoA  by  requiring  that  all  acquisition  programs: 

Evaluate  the  cost,  schedule,  and  performance  of  the  program, 
relative  to  current  metrics,  performance  requirements,  and  baseline 
parameters,  and  determine  the  extent  to  which  the  level  of  program 
cost,  schedule,  and  performance,  relative  to  established  metrics,  is 
likely  to  result  in  the  timely  delivery  of  a  level  of  capability  to  the 
Warfighter  that  is  consistent  with  the  level  of  resources  to  be 
expended  and  to  provide  superior  value  to  alternative  approaches 
that  may  be  available  to  meet  the  same  requirement.  (Hansen, 

2010) 
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In  performing  AoA  studies,  the  Navy  and  Air  Force  identify  trade-off 
analysis  as  a  key  factor  for  HSI  involvement.  The  Navy  Acquisition  gate  process 
(Whitaker,  2009)  (See  Figure  7)  identifies  capability  gaps  and  applies  acceptable 
operational  thresholds  and  objectives  when  conducting  the  trade-off  studies.  At 
the  Gate  2  review,  analysis  of  DOTMLPF  must  occur.  The  HSI  practitioner  can 
contribute  by  assessing  tradeoffs  among  HSI  domains  and  providing  specific  cost 
adjustments  to  mitigate  cost  overruns.  Practitioners  also  identify  capability  gaps 
to  provide  input  for  rough  order  of  magnitude  cost  assessment  as  specified  in 
OPNAV  Instruction  5310.23.  Capability  gap  contributions  and/or  limitations  may 
include: 

•  Manpower,  personnel,  or  training  (including  military,  contractor, 
civilian  and  recruitment/retention  of  each), 

•  Contributions  of  operator,  maintainer,  support  personnel,  total 
system  performance  (including  infrastructure), 

•  Requirements  for  safety,  survivability,  habitability,  and 

•  System  reliability,  maintainability,  transportability,  and  supportability 

A  goal  of  capability  gap  identification  is  the  reduction  of  the  system’s  logistics 
footprint  and  total  ownership  cost  (Shattuck,  2010). 
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Acquisition  Governance  Board  Tiger  Team  Draft  Gate  Alignment  2.0 
as  of  08/03/09  DODI  5000.02  (Dec  2008) 
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Figure  7.  Navy  Gate  Alignment  Dec  2008  (From  Whitaker,  2009) 


The  aforementioned  gap  analyses  are  an  important  exit  criteria  for 
the  materiel  solution  analysis  phase  and  are  assessed  at  Milestone  A  prior  to  the 
development  of  the  system.  Furthermore,  the  use  of  the  capabilities  assessments 
will  allow  comparisons  with  benchmark  resources  (current  system  capabilities 
and  costs)  and  future  developments  in  AoA. 

The  Human  System  Engineering  (HSE)  Manual  provides  further 
guidance  for  HSI  activities  and  cost  considerations  during  the  MSA  phase. 
Insights  into  potential  COTS  materiel  solution  approaches  and  HSI  domain  trade- 
space  analysis  are  specified.  The  HSE  Manual  states: 

The  COTS  approach  has  (a)  the  lowest  projected  life  cycle  cost, 
within  acceptable  risks,  and  meets  essential  requirements, 
including  human  performance  requirements;  (b)  the  lowest  human 
workload  and  manning  requirements  for  operations  and 
maintenance;  (c)  the  most  effective  training  program;  and  (d)  the 
least  safety  and  health  hazards.  (Naval  Sea  Systems  Command, 

2009,  p.  29) 
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Domain  trade-offs  include  optimizing  manpower,  addressing 
training  requirements,  incorporating  safety,  and  designing  for  survivability.  In 
conducting  trade-offs,  manpower  can  be  optimized  by  including  the  proper  mix  of 
personnel,  conducting  a  workload  analysis,  reviewing  policy,  and  increasing 
performer  effectiveness.  Training  requirements  for  specific  designs  are  also 
considered.  Risk  reduction  and  hazard  prevention  in  system  design  must 
consider  safety  and  health  concerns.  In  trade-off  studies,  the  personnel 
survivability  equipment  design  is  an  essential  consideration.  Increased 
performance  and  maintenance  requirements  are  used  to  research  and  test 
equipment  design.  The  Life  Cycle  Sustainment  Plan  (LCSP)  also  recognizes 
these  domain  initiatives.  The  LCSP  considers  the  domain  trade-offs  as 
contributing  factors  to  the  reduction  of  total  operating  costs. 

Finally,  habitability  requirements  for  personnel  are  considered,  but 
generally  are  the  least  found  in  trade-off  studies.  Habitability  trade-offs  may 
include  design  of  living  conditions  that  would  impact  working  conditions  and 
quality  of  life.  Thus  far,  section  (a)  has  considered  HSI  activity  development  and 
trade-off  studies  in  the  MSA  phase.  The  next  step  concerns  the  technical  review 
process  that  ensures  performance  and  funding  requirements  are  met. 

The  MSA  phase  includes  two  technical  review  processes  to  ensure 
programs  meet  performance  and  funding  requirements.  These  technical  reviews 
are  the  Initial  Technical  Review  (ITR)  and  the  Alternative  System  Review  (ASR). 
The  ITR  ensures  cost  estimate  validity,  assesses  capability  needs  and  materiel 
solutions,  and  identifies  cost  drivers.  As  stated  in  the  NAVSEA  HSE  manual,  the 
ITR  process  may  include  HSI  assessments  based  on  mission  scenario  functions 
allocated  by  the  Navy  Tactical  Task  list.  By  doing  so,  performance  metrics  are 
identified  and  thus,  drivers  (i.e.,  manpower  effort)  for  cost  relationships  are 
established.  ITR  scenarios  may  also  identify  training  requirements  imposed  by 
system  operation. 
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The  ASR,  on  the  other  hand,  ensures  that  the  system  is  ready  to 
move  on  to  the  Technology  Development  phase  and  meets  the  requirements  set 
by  the  user.  The  ASR  will  provide  a  review  of  the  AoA  and  provide  a 
comprehensive  rationale  for  the  proposed  solution  (Shattuck,  2010).  By 
reviewing  alternative  materiel  solutions,  the  ASR  helps: 

...  ensure  that  sufficient  effort  has  been  given  to  conducting  trade 
studies  that  consider  and  incorporate  alternative  system  designs 
that  may  more  effectively  and  efficiently  meet  the  defined 
capabilities.  (DAU  Online,  2010) 

The  HSI  practitioner  aids  in  this  process  by  being  involved  in  the  Integrated 
Product  Team  and  by  providing  input  for  workload  and  manpower  requirements 
for  each  alternative.  By  doing  so,  performance  effectiveness,  training 
assessments,  system  interoperability,  test  and  evaluation  (T&E)  and  life  cycle 
costs  (including  schedule,  cost,  and  performance  trade-offs)  for  each  alternative 
reflects  actual  costs  more  accurately. 

The  Materiel  Solution  Analysis  phase  ends  when  the  exit  criteria 
are  met,  including: 

•  Initial  acquisition  strategy  (including  the  sustainment  strategy), 

•  Contractual  documents  required  to  continue  into  the  Technology 
Development  Phase, 

•  Initial  support  and  maintenance  concepts, 

•  Life  cycle  costs,  and 

•  Manpower  estimates  for  the  system  concept. 

Finally,  input  is  provided  for  the  draft  CDD  and  the  Test  &  Evaluation  Strategy 
(TES).  The  TES  must  include  estimated  costs  pertaining  to  system  operation  and 
system  attributes.  HSI  analysis  will  be  included  in  the  TES  and  must  be  included 
in  cost  estimates. 
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Section  (a)  provided  an  overview  of  HSI  activities  in  the  Materiel 
Solution  Analysis  phase.  See  Table  1  for  a  summary  of  HSI  activities  during  the 
MSA  phase.  HSI  manpower  effort  and/or  HSI  funding  requirements  were 
established.  Section  (b)  will  provide  an  overview  of  the  same  issues  for  the 
Technology  Development  phase. 


Table  1 .  MSA  Phase  HSI  Activity  Summary 


HSI  Activity 

Publication 

IDAT&L  Activity 

1.  Determine  issues  related  to 
physical/mental  characteristics. 

SECNAVM-5000.2 

Technology  Development 
Strategy  (TDS) 

2.  Describe  the  target  audience  description 
(TAD). 

SECNAVM-5000.2 

TDS 

3.  Provide  performance  parameters  for 
operators  and  maintainers. 

SECNAVM-5000.2 

TDS 

4.  Assess  affordability,  feasibility  and  risk. 
Consider  training,  manpower,  job 
specialization  (Personnel  Rates/MOS),  and 
technology  insertion  (Enclosure  7). 

SECNAVM-5000.2 

TDS 

5.  Conduct  scenario  and  threat  development 
analysis  as  well  as  capability  ranges. 

Defense 

Acquisition 

Guidebook 

AoA  Study  Plan 

6.  Assess  the  critical  technology  elements. 

DoD  5000  Series 

AoA 

7.  Review  DOTMLPF. 

8.  Conduct  trade-off  analysis  of  among  HSI 
domains  and  provide  specific  cost 
adjustments  to  mitigate  cost  overruns. 

2010  WSARA 

AoA 

9.  Identify  capability  gaps  to  provide  input  for 
rough  order  of  magnitude  cost  assessment. 

OPNAV  Instruction 
5310.23 

AoA 

10.  Conduct  a  COTs  HSI  domain  trade-space 
analysis. 

HSE  Manual 

AoA 

11.  Review  HSI  mission  scenario 
assessments,  metrics,  training  requirements 

HSE  Manual 

Initial  Technical  Review 
(ITR) 

12.  Participate  in  Integrated  Product  Team; 
Provide  input  for  workload  and  manpower 
requirements  for  each  alternative. 

Shattuck,  2009 

Alternative  System  Review 
(ASR) 

13.  Provide  input  for  draft  CDD  and  the  Test 
and  Evaluation  Strategy  (TES)  pertaining  to 
system  operation  and  system  attributes. 

Shattuck,  2009 

Alternative  System  Review 
(ASR) 
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b.  Technology  Development  Phase 

The  focus  of  this  phase  is  ‘to  reduce  technology  risk  and  to 
determine  the  appropriate  set  of  technologies  to  be  integrated  into 
the  full  system.  This  effort  is  normally  funded  only  for  advanced 
development  work  and  does  not  mean  that  a  new  acquisition 
program  has  been  initiated’.  (DAU  Online,  2010) 

DoDI  5000.02  and  the  JCIDS  Manual  (2009)  state  the  overall  goal 
of  the  Technology  Development  (TD)  phase  is  to  reduce  risk  and  determine  the 
maturity  of  systems  for  full  system  integration  and  prototype  development.  The 
TD  phase  contributes  to  continuous  technology  discovery  and  successful 
collaboration  between  the  user,  the  S&T  community,  and  the  system  developer. 
Viability  of  technologies  and  refining  user  requirements  is  of  utmost  importance. 
During  this  phase,  inputs  are  required  for  the  following  documents  and  activities: 

•  Capability  Development  Document  (CDD), 

•  Acquisition  Strategy  (AS), 

•  Acquisition  Program  Baseline  (APB), 

•  System  Engineering  Plan  (SEP), 

•  Test  and  Evaluation  Master  Plan  (TEMP), 

•  Request  for  Proposal  (RFP), 

•  System  Performance  Specifications,  and 

•  Technical  Reviews 

In  this  section,  the  following  resources  will  be  used  to  describe  the 
TD  phase  HSI  activities: 

•  DoD  5000  series  instruction, 

•  JCIDS  Manual, 

•  OPNAVINST  5310.23, 
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•  711**^  Human  Performance  Wing  Human  System  Integration 

Development  (HPW  HSI)  Guide, 

•  Human  Systems  Engineering  (HSE)  Manual,  and 

•  Secretary  of  the  Navy  (SECNAV)  guidance 

HSI  practitioner  involvement  in  the  TD  phase  begins  with  the 
development  of  the  Capability  Development  Document  (CDD).  DoDI  5000.02 
states  that  every  acquisition  program  or  evolutionary  increment  requires  a  CDD. 
The  CDD  shall  refine  integrated  architecture  as  well  as  clarify  how  the  program 
will  lead  to  joint  warfighting  capability.  Additionally,  the  JCIDS  Manual  states  that 
the  CDD  must  address  the  system  capabilities  and  key  performance  parameters 
(KPPs)  of  the  system.  HSI  issues  that  affect  the  suitability,  affordability  and 
effectiveness  of  the  system  are  also  addressed  (Department  of  Defense,  2008c). 
The  JCIDS  manual  specifies  that  design  considerations  must  consider 
environmental  factors,  unplanned  stimuli  and  mission  capability.  Mission 
capability  is  extremely  important  especially  in  nuclear,  biological,  chemical,  and 
environments. 

OPNAVINST  5310.23  describes  the  HSI-related  capabilities  that 
should  be  discussed  in  the  CDD.  The  KPPs  should  include  threshold  and 
objective  values  that  ensure  Warfighters  are  able  to  achieve  required  levels  of 
performance.  The  711*^^  HPW  HSI  Guide  and  HSE  Manual  address  the  human- 
centered  KPPs  and  capabilities  that  should  be  considered  in  system  design. 

HSI  practitioners  also  provide  input  into  the  Acquisition  Strategy 
(AS)  and  Acquisition  Program  Baselines  (APBs).  HSI  input  (as  part  of  an  IPT)  is 
governed  by  SECNAV  guidance  and  service  policies.  SECNAVINST  5000. 2D 
indicates: 

Life  cycle  cost  projections  for  capabilities  and/or  systems  shall 
include  direct  HSI  costs  (e.g.,  MPT),  and  should  discuss  indirect 
costs  (e.g.,  medical  benefits  resulting  from  safety  and  occupational 
health  risks)... HSI  issues  and  domains  must  be  considered  to 
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ensure  that  configuration  changes  do  not  create  new  or  unforeseen 
HSI  issues.  (Department  of  Defense,  2008d,  p.  12) 


Milestone  B  approval  is  dependent  on  the  Acquisition  Strategy  which  will 
incorporate  development,  testing,  production  and  life  cycle  support  as  critical 
elements. 

APBs  are  a  necessary  step  in  establishing  the  trade-offs  between 
cost,  schedule  and  performance.  Many  of  the  KPP,  KSA,  threshold/objective 
values,  and  requirements  are  human  performance  based,  which  affect  system 
performance.  Within  the  APB,  operationally  determined  performance  parameters, 
measures  of  effectiveness,  and  performance  help  to  determine  drivers  cost 
estimates.  The  APB  consists  of  the  following  components  (see  Table  2): 


Table  2.  APB  Cost  Involvement  (From  Shattuck,  2009) 


Research,  development,  test  and 
evaluation  costs 

Operations  and  maintenance  (O&M) 
costs 

Procurement  costs 

Average  Procurement  Unit 

Military  construction  costs 

Program  Acquisition  Unit  Cost 

Total  system  quality 

Any  other  cost  objectives  established  by 
the  milestone  Decision  Authority 

The  APB  is  approved  by  the  MDA  and  must  concur  with  the  Program  Executive 
Officer  (PEO)  for  all  ACAT  programs,  and  the  DoD  Component  Acquisition 
Executive  (CAE)  for  ACAT  ID  and  lAM  programs  (Shattuck,  2009). 

The  HSE  Manual  describes  the  application  of  HSI  initiatives  in  the 
Acquisition  Strategy,  System  Engineering  Plan,  Test  and  Evaluation  Master  Plan 
and  Request  for  Proposal.  To  fulfill  the  HSI  objectives  of  the  system,  the  manual 
incorporates  the  following: 

•  Human  performance, 

•  Workload, 

•  Safety  issues. 
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•  Risk  assessments,  and 

•  Reduction  in  technology  applications 

HSI  practitioners  also  provide  input  on  human  techniques,  system  interactions 
(i.e.,  software,  hardware)  and  HSI  metrics.  The  activities  will  emphasize  “mission 
capability,  readiness,  force  structure,  affordability,  performance  effectiveness, 
and  achievement  of  wartime  operational  objectives”  (NAVSEA,  2009,  pp.  6-51). 
By  doing  so,  HSI  needs  and  requirements  are  successfully  included  in 
technology  contracts  and  prototype  RFPs. 

A  major  output  of  the  TD  phase  is  the  system  performance 
specifications.  HSI  practitioners  provide  input  to  the  program  manager  to  help 
determine  trade-offs  for  cost,  schedule  and  performance.  The  HSE  Manual 
addresses  performance  specifications  by  identifying  human  roles  and 
responsibilities.  The  manual  includes  “Requirements  for  sustained  human 
performance... prevention  of  human  error... approaches  that  reduce  human  error 
and  cognitive  workload  and  human  machine  interface  designs  that  facilitate 
human  performance”  (Shattuck,  2009,  p.4;  NAVSEA,  2009,  pp.6-55). 

Finally,  three  major  reviews  occur  in  the  TD  phase.  These  reviews 
are  the  System  Requirement  Review  (SRR),  System  Functional  Review  (SFR) 
and  the  Preliminary  Design  Review  (PDR).  HSI  input  into  each  of  these  reviews 
allows  for  the  mitigation  of  potential  shortfalls  and  the  survivability  of  the  system 
in  future  operational  testing,  evaluation  and  verification  events. 

The  SRR  specifies  the  allocation  of  tasks  to  the  human,  software, 
and  hardware  in  the  system.  The  SFR  includes  operational  level  functional 
requirements  for  the  various  subsystems.  The  review  also  allows  for  assignment 
of  those  requirements  to  operations,  maintenance  and  sustainment 
considerations.  IPTs  can  then  initiate  preliminary  design. 

The  PDR  establishes  a  “physically  allocated  baseline”  that  helps 
determine  the  level  of  operational  effectiveness  that  the  system  will  demonstrate 
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(Shattuck,  2009).  The  PDR  ensures  design  configuration  and  capabilities  are  in 
accordance  with  cost,  schedule,  and  performance  requirements.  The  PDR  may 
be  completed  in  the  Engineering  and  Manufacturing  Development  (E&MD) 
phase.  The  TD  phase  ends  at  Milestone  B  and  at  the  completion  of  the 
preliminary  design. 

Section  (b)  provided  an  overview  of  HSI  activities  in  the  Technology 
Development  phase.  Table  3  provides  a  summary  of  the  HSI  activities  during  the 
TD  phase.  Section  (c)  will  provide  an  overview  of  the  same  issues  for  the 
Engineering  and  Manufacturing  Development  phase. 
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Table  3.  TD  Phase  HSI  Activity  Summary 


HSI  Activity 

Publication 

IDAT&L  Activity 

1.  Address  the  system  capabilities 
and  key  performance  parameters 
(KPPs). 

2.  Assess  the  suitability,  affordability 
and  effectiveness  of  the  system 
(consider  environmental  factors, 
unplanned  stimuli  and  mission 
capability). 

DoDI  5000.02; 
JCIDS  Manual; 
OPNAVINST 
5310.23;  71 1‘^ 
HPW  HSI  Guide 
and  HSE 

Manual 

Capability 

Development 

Document  (CDD), 

3.  Life  cycle  cost  projections  for 
capabilities  and/or  systems  shall 
include  direct  HSI  cost. 

4.  HSI  issues  and  domains  must  be 
considered  to  ensure  that 
configuration  changes  do  not  create 
new  or  unforeseen  HSI  issues. 

SECNAVINST 
5000.2D; 
service  policies 

Acquisition  Strategy 
(AS), 

5.  Assess  KPP,  KSA, 
threshold/objective  values,  and 
requirements  for  trade-off  analysis. 

SECNAVINST 
5000.2D; 
service  policies 

Acquisition  Program 
Baseline  (APB), 

6.  Assess  system  for  human 
performance,  workload,  safety  issues, 
risk  assessments,  and  reduction  in 
technology  applications. 

7.  Provide  input  on  human 
techniques,  system  interactions  (i.e., 
software,  hardware)  and  HSI  metrics. 

HSE  Manual 

System  Engineering 
Plan  (SEP),  Test  and 
Evaluation  Master 

Plan  (TEMP),  and 

8.  Assess  HSI  needs  and 
requirements  in  the  technology 
contracts. 

HSE  Manual 

Request  for  Proposal 
(RFP). 

9.  Provide  input  to  the  PM  to 
determine  trade-offs  for  cost, 
schedule  and  performance. 

HSE  Manual 

System  performance 
specifications 

10.  Provide  input  into  the  technical 
reviews  for  mitigation  of  potential 
shortfalls  and  the  survivability  of  the 
system  in  future  operational  testing, 
evaluation  and  verification  events. 

HSE  Manual 

System  Requirement 
Review  (SRR), 

System  Functional 
Review  (SFR)  and  the 
Preliminary  Design 
Review  (PDR). 
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c. 


Engineering  and  Manufacturing  Deveiopment  Phase 


The  purpose  of  the  E&MD  Phase  is  to  develop  a  system  or  an 
increment  of  capability;  complete  full  system  integration;  develop  an 
affordable  and  executable  manufacturing  process;  ensure 
operational  supportability  with  particular  attention  to  minimizing  the 
logistics  footprint;  implement  human  systems  integration  (HSI); 
design  for  producibility  and  ensure  affordability;  protect  CPI  by 
implementing  appropriate  techniques  such  as  anti-tamper;  and 
demonstrate  system  integration,  interoperability,  safety,  and  utility. 

(DAU  Online,  2010) 

The  Engineering  and  Manufacturing  (E&MD)  phase  consists  of  two 
efforts:  the  Integrated  System  Design  (ISD)  and  the  System  Capability  and 
Manufacturing  Process  Demonstration  (SC&MPD).  The  E&MD  phase  begins 
after  a  successful  Milestone  B  review  and  includes  a  Post  Critical  Design  Review 
(CDR)  Assessment  at  the  end  of  the  ISD.  The  Post-CDR  Assessment  will 
confirm  design  maturity  and  provide  the  initial  product  baseline.  The  following 
P&D  phase  documents  guide  the  E&MD  phase: 

•  Capability  Development  Document, 

•  Acquisition  Strategy, 

•  System  Engineering  Plan,  and 

•  Test  Evaluation  Master  Plan  (TEMP) 

The  SC&MPD  effort,  completed  after  the  Post-Critical  Design  Review,  shall 
demonstrate  the  operation  of  the  systems  (with  the  approved  KPPs)  in  the 
intended  environment  of  use. 

The  E&MD  phase  contains  six  embedded  processes  (i.e.,  JCIDS, 
Oversight  and  review.  Contracting,  Major  Products,  Logistics/  Sustainment  and 
Technical).  HSI  practitioners  are  not  directly  involved  in  production  and 
manufacturing  activities;  however,  practitioners  do  monitor  the  activities  to 
reduce  risk.  The  initial  phase  activity  includes  MDA  updates  to  the  Acquisition 
Strategy  and  APB  administrative  review  processes  that  should  contain  HSI  input. 
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This  Input  ensures  that  a  defense  acquisition  program  continues  to  have  an  HSI 
emphasis  and  that  program  managers  will  be  able  to  maintain  planned 
schedules.  HSI  considerations  are  important  as  the  E&MD  continues  with  the 
Source  Selection  Evaluation  Board  (SSEB).  HSI  board  members  will  review 
RFPs  for  HSI  concerns.  A  majority  of  HSI  involvement  occurs  during  prototype 
development. 

In  prototype  development,  HSI  practitioners  help  to  build  and  use 
the  models  and  simulation  tools  for  testing  purposes.  By  doing  so,  human 
performance  modeling  allows  mitigation  of  risk  and  safety  issues  that  can  occur 
early  in  the  system  development.  Further  HSI  activities  revolve  around  the 
Product  Support  Plan  (PSP)  and  the  Performance-Based  Logistics  (PBL) 
Strategy.  These  activities  focus  on  several  HSI  domains:  Manpower,  Training, 
Personnel,  HFE  and  Habitability.  Because  the  system  may  change  operationally, 
by  mission  capability,  or  by  change  in  organization  and/or  funding,  the  logistician 
must  update  the  PSP  repeatedly.  Therefore,  HSI  practitioners  are  continuously 
required  to  provide  input.  The  DoDI  5000.02,  DAG  (2010)  and  Shattuck  (2009) 
state  that  these  inputs  to  include: 

•  Reduction  of  time  in  maintenance  and  /  or  operation  due  to  HFE 
influenced  design, 

•  Requirement  development  during  Personnel  Selection  (i.e., 
knowledge,  skills,  attribute,  and  experience  levels), 

•  Manpower  reduction  or  increase  use  of  the  proper  mix  of  military, 
contract,  civilian  workers, 

•  Training  development,  test  evaluation  and  verification, 

•  Habitability  concerns  such  as  “morale,  safety,  health,  comfort, 
personnel  performance,  or  unit  readiness,  that  could  lead  to 
recruitment  or  retention  problems”  (Shattuck,  2009),  and 
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•  Environment,  Safety  and  Operational  Health  concerns  such  as 
mitigation  of  health  deterioration  or  mishaps  due  to  system  design 
and  human  centered  environmental  designs. 

While  habitability  is  important  in  design  considerations,  the  domain 
criteria  usually  fall  under  other  guidance.  References  to  publications  that  guide 
habitability  design  efforts  include:  (a)  the  Naval  Sea  Systems  Command  (2006) 
Joint  Fleet  Maintenance  Manual  (COMFLTFORCOMINST  4790.3),  and  (b)  the 
Air  Force  Air  Force  Habitability  and  Occupational  Health  publications.  These 
publications  are  service  and/or  system  specific  and  refer  to  the  following 
guidance: 

•  OPNAVINST  9640.1,  Shipboard  Habitability  Program, 

•  NAVSEA  SL720-AA-MAN-010/V1R2.  Fleet  Modernization  Program 
(FMP)  Management  And  Operations  Manual 

•  NAVSEA  SL720-AA-MAN-030.  Surface  Ship  and  Carriers  Entitled 
Process  for  Modernization  Management  and  Operations  Manual 

•  Air  Force  Policy  Directive  23-1  (2009),  Materiel  Management 
Policies  and  Procedures, 

•  Air  Force  Policy  Directive  21-1  (2003),  Air  and  Space  Maintenance, 

•  Air  Force  Occupational  Safety  And  Health  Standard  48-20  (2006), 
Occupational  Noise  And  Hearing  Conservation  Program, 

•  Air  Force  Handbook  21-130,  Technical  Analysis  To  Determine 
Criterion  For  2  Vs  3  Level  Repair, 

•  Air  Force  Occupational  Safety  And  Health  Standard  48-139  (1999), 
Laser  Radiation  Protection  Program, 

•  Air  Force  Pamphlet  90-902  (2000),  Operational  Risk  Management 
(ORM)  Guidelines  And  Tools, 

•  Preliminary  Hazard  Analysis,  Hazard  Listing,  Risk  Assessment,  and 
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•  Air  Force  Instruction  90-1301(2008),  Implementing  Military  Flight 
Operations  Quality  Assurance. 

The  CPD  is  one  of  the  final  documents  produced  during  the  E&MD 
phase.  The  CPD  helps  to  determine  the  operational  performance  parameters. 
The  CPD  is  revised  as  lessons  are  learned  during  system  development  and  as 
KPPs  and  KSAs  are  refined.  The  ICD,  CDD,  AoA,  CDR,  development  and 
operational  testing,  DoD  Enterprise  Architecture,  and  DOTMLPF 
recommendations  also  influence  the  CPD.  Once  approved  by  the  JROC  for 
operational  acceptance,  the  MDA  can  initiate  Milestone  C  and  approve  entry  into 
the  Production  and  Deployment  (P&D)  phase. 

Military  service  guidance  for  incorporating  HSI  requirements  into 
the  CPD  include  the  (a)  Navy  OPNAVINST  5310.23,  (b)  Air  Force  71 1‘^  HSI 
Development  Guide,  and  the  (c)  Army  MANPRINT  documents.  Examples  of  HSI 
requirement  activities  include: 

•  Identify  the  integrated  training  system  requirements  for  individual, 
collective,  joint,  and  fleet  training  support  fe.g..  Total  Ship  Training 
System  for  ships). 

•  Describe  in  measurable  and  testable  terms,  when  relevant,  the 
missions,  functions,  or  attributes  used  to  optimize  manpower, 
personnel  readiness  requirements. 

•  Identify  MPT  Total  Ownership  Cost  (TOC)  savings  achieved 
through  analysis  of  design  (human  factors,  workload,  KSAs, 
resources,  and  tools)  for  all  tasks  allocated  to  humans  required  for 
the  efficient  operation,  employment,  and  support  of  the  system. 

•  Describe  any  safety,  occupational  health,  and  environmental 
compliance  requirements  that  reduce  the  risk  of  fatalities,  injury, 
illness,  or  disability  and  death  of  the  operators,  maintainors,  and 
support  personnel. 
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•  Describe  habitability  requirements  for  facilities  (berthing,  personal 
stowage,  food  service,  medical,  religious,  security,  recreational  and 
lounge  spaces),  and  ambient  environment  requirements  (e.g., 
noise,  lighting,  heating,  air  conditioning,  and  ventilation,  workspace 
layout,  etc.)  (Department  of  Defense,  2009e) 

Finally,  several  System  Engineering  (SE)  Reviews  occur  prior  to 
exiting  the  E&MD  phase.  The  SE  Reviews  include  the  Test  Readiness  Review 
(TRR),  the  Production  Readiness  Review  (PRR),  the  System  Verification  Review 
(SVR),  and  the  Functional  Configuration  Audit  (FCA).  In  all,  the  reviews  are  in 
place  to  ensure  the  designed  system  meets  the  specified  requirements  and 
capabilities  but  also,  to  allow  Low  Rate  Initial  Production  and  Full  Rate 
Production  to  occur.  As  members  of  an  Integrated  Product  Team,  FISI 
practitioners  are  fundamental  to  the  success  of  these  reviews.  MSI  input  will 
ensure  the  designs  are  functional  from  the  users’  perspective  with  minimal  risk  to 
them. 

The  Technical  Outputs  are  similar  to  those  in  the  Technology 
Development  phase;  however,  an  Initial  Product  Baseline  (IPB)  is  required  for  all 
class  one-configuration  changes.  As  stated  previously,  IPTs  are  required  to 
incorporate  into  the  IPB  the  following: 

Functional  and  physical  characteristics  of  the  Configuration  Item, 
any  joint  and  combined  operations  that  the  configuration  item, 
functional  and  physical  characteristics  used  for  production 
acceptance  testing,  and  tests  that  are  required  for  the  deployment, 
installation,  support,  training,  and  disposal  of  the  Configuration 
Item.  (Defense  Acquisition  University  (DAU),  2010) 

Section  (c)  provided  an  overview  of  MSI  activities  in  the 
Engineering  and  Manufacturing  Development  phase.  See  Table  4  for  a  summary 
of  MSI  activities  during  the  E&MD  phase.  Section  (d)  will  provide  an  overview  of 
the  same  issues  for  the  Production  and  Deployment  phase. 
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Table  4.  E&MD  Phase  HSI  Activity  Summary 


HSI  Activity 

Publication 

IDAT&L  Activity 

1.  Monitor  activities  to  reduce  risk. 

DoDI  5000.02,  DAG  (2010) 
and  Shattuck  (2009) 

Production  and 

Manufacturing  activities 

2.  Review  processes  for  HSI 
emphasis  for  MDA  updates. 

DoDI  5000.02,  DAG  (2010) 
and  Shattuck  (2009) 

Acquisition  Strategy  and 

APB 

3.  Provide  HSI  board  members  for 
RFP  reviews. 

DoDI  5000.02,  DAG  (2010) 
and  Shattuck  (2009) 

Source  Selection  Evaluation 
Board  (SSEB) 

4.  Perform  Modeling  and  Simulation 
activities:  human  performance 
modeling;  mitigation  of  risk  and 
safety  issues. 

DoDI  5000.02,  DAG  (2010) 
and  Shattuck  (2009) 

Prototype  Development 

5.  Provide  Manpower,  Training, 
Personnel,  HFE  and  Habitability 
updates  due  to  mission  capability, 
organization  or  funding  change. 

DoDI  5000.02,  DAG  (2010) 
and  Shattuck  (2009); 
service  policies 

Product  Support  Plan  (PSP) 
and  the  Performance-Based 
Logistics  (PBL)  Strategy 

6.  Identify  Training  requirement; 
Optimize  Manpower/Personnel 
readiness  requirements;  Identify 

MPT  Total  Ownership  Cost  (TOC); 
Describe  ESOH  requirements; 
Describe  Habitability  Requirements. 

OPNAVINST  5310.23,  Air 
Force  711"^  HSI 

Development  Guide;  Army 
MANPRINT  documents 

Capability  Production 
Document  (CPD) 

7.  Provide  HSI  input  to  ensure  the 
designs  are  functional  from  the 
user’s  perspective  with  minimal  risk 
to  them. 

OPNAVINST  5310.23,  Air 
Force  7 11"^  HSI 

Development  Guide;  Army 
MANPRINT  documents 

System  Engineering 

Reviews:  Test  Readiness 
Review  (TRR),  the 

Production  Readiness 

Review  (PRR),  the  System 
Verification  Review  (SVR), 
and  the  Functional 
Configuration  Audit  (FCA). 

8.  Participate  in  an  IPT  to  provide 
functional  and  physical 
characteristics  for  the  configuration 
Items. 

OPNAVINST  5310.23,  Air 
Force  7 11"^  HSI 

Development  Guide;  Army 
MANPRINT  documents 

Technical  Outputs:  IPB; 

Same  as  TD  phase 

d.  Production  and  Deployment  Phase 

The  overall  focus  of  the  Production  and  Deployment  phase  is  to 
‘complete  the  development  of  a  system  or  increment  of  capability, 
leveraging  design  considerations;  complete  full  system  integration; 
develop  an  affordable  and  executable  manufacturing  processes, 
complete  system  fabrication,  test  and  evaluation’.  (DAU  Online, 
2010) 


The  purpose  of  the  Production  and  Deployment  (P&D)  phase  is  to 
achieve  an  operational  capability  that  satisfies  the  mission  need.  This  phase 
consists  of  two  efforts:  Low  Rate  Initial  Production  (LRIP)  and  Full  Rate 
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Production  and  Deployment  (FRP&D).  The  phase  begins  after  a  successful 
Milestone  C  review.  The  Full  Rate  Production  Decision  Review  (FRPDR) 
separates  the  two  efforts.  MSI  practitioners  assist  in  these  efforts  by  participating 
in  integrated  product  teams  and  providing  updates  to  documentation. 

Three  reviews  take  place  at  the  beginning  of  the  Production  and 
Deployment  phase.  These  reviews  include  the  Operational  Test  Readiness 
Review  (OTRR),  the  Assessment  of  Operational  Test  Readiness  (AOTR),  and 
the  Physical  Configuration  Audit  (PCA).  The  OTRR  is  required  prior  to  Initial 
Operational  Test  and  Evaluation  (lOT&E),  which  assesses  the  operational 
capabilities,  suitability  and  effectiveness  of  the  product.  The  AOTR  occurs  upon 
demonstration  of  those  operational  capabilities.  The  PCA  determines  if: 

Design  documentation  matches  the  item  as  specified  in  the 
contract;  determines  that  the  manufacturing  processes,  quality 
control  system,  the  measurement  and  test  equipment  as  well  as 
training  programs  are  planned,  tracked,  and  controlled;  and  defines 
the  starting  point  for  controlling  the  detail  design  and  establishing  a 
product  baseline.  (Shattuck,  2009) 

MSI  participation  in  these  reviews  is  essential.  Each  review  employs  a  checklist 
that  is  used  to  evaluate  the  systems.  The  checklists  include  activities  such  as 
Program  Management,  Logistics,  Testing,  Interoperability,  etc.  The  OTRR 
checklist,  for  example,  contains  96  FISI  evaluation  activities  (out  of  a  total  of  143 
items).  The  PCA  contains  143  FISI  items. 

MSI  is  also  present  in  the  Defense  Acquisition  Program  Support 
(DAPS)  Methodology  for  System  and  Software  Engineering.  The  methodology 
provides  examples  of  MSI  domain  activities  and  provides  criteria  to  evaluate  the 
activities.  Although  the  reference  is  not  used  extensively  in  this  thesis  to  identify 
the  MSI  activities  for  the  cost  estimation  process,  it  should  be  considered  for 
future  research.  A  DAPS  example  of  the  manpower  domain  guidance  is  as 
follows: 
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Manpower  factors  include  job  tasks,  operation  and  maintenance 
(O&M)  rates,  workload,  and  operational  conditions  used  to 
determine  the  number  and  mix  of  military  and  DoD  civilian 
manpower  needed  to  operate,  support,  maintain  and  provide 
training  for  the  system. 

DoDD  5000.1  requires  the  Component  Services  to  plan  programs 
within  projected  future  year  manpower  availability.  Program 
manpower  requirements  should  be  based  on  studies  and  analyses 
that  consider  all  operational  facets  of  the  Concept  of  Operations 
(CONORS)  to  account  for  the  manpower  mix,  and  the  impact  on 
any  established  Service-level  constraints  on  manpower  end 
strength.  (Department  of  Defense,  2000a) 

Other  HSI  activities  in  the  P&D  phase  include  updating  the  Product 
Support  Plans  (PSP)  and  verifying  Warfighter  requirements  in  the  Pre-Initial 
Operational  Capability  (IOC)  review.  As  stated  previously,  HSI  practitioners  are 
not  directly  involved  in  writing  the  PSP  but  work  with  logisticians  to  update  HSI 
criteria  (requirements  and  performance  metrics  use  to  optimize  the  system) 
within  the  document.  The  IOC  review  updates  configuration  control  changes, 
notes  deficiency  correction  statuses  within  the  system,  and  defines  the  maturity 
of  the  system. 

Two  of  the  most  important  aspects  of  the  P&D  phase  are  the  Initial 
Operational  Test  and  Evaluation  (lOT&E)  and  the  Live  Fire  Test  and  Evaluation 
(LFT&E)  activities.  HSI  practitioners  (as  part  of  IPTs)  help  to  identify  testable  HSI 
metrics  for  these  tests.  OT&E  activities  occur  prior  to  Low-Rate  Production 
decisions.  The  OT&E  ensures  that  suitability  and  effectiveness  meets  the 
requirements  of  the  “Warfighter,  government  civilians,  and  DoD  contractors  that 
will  operate,  maintain,  and  support  the  system,  and  those  who  will  train  them” 
(Shattuck,  2009).  The  LFT&E  tests  for  the  vulnerability  or  lethality  of  the  system. 
HSI  is  indirectly  involved  in  this  testing;  however,  practitioners  can  learn  how  the 
system  design  affects  the  susceptibility  and  recoverability  of  the  system,  thus 
mitigating  safety  and  mishap  concerns. 
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Finally,  scheduling  or  funding  cuts  may  remove  HSI  requirements. 
Therefore,  prior  to  the  FRP,  the  Acquisition  Strategy,  Acquisition  Plan,  Source 
Selection  Plan,  RFPs  and  Final  Product  Baseline  documents  receive  FISI  criteria 
updates.  This  is  the  last  line  of  defense  in  advocating  for  the  representative  users 
of  the  product.  These  documents,  as  well  as  manpower  estimations,  economic 
analyses.  Critical  Safety  List  items,  PESFIE  and  TEMP,  will  be  reviewed  at  the 
FRP  by  the  MDA. 

Section  (d)  provided  an  overview  of  HSI  activities  in  the  Production 
and  Deployment  phase.  See  Table  5  for  a  summary  of  HSI  activities  during  the 
P&D  phase.  Section  (e)  will  provide  an  overview  of  the  same  issues  for  the 
Operation  and  Support  phase. 


Table  5.  P&D  Phase  HSI  Activity  Summary 


HSI  Activity 

Publication 

IDAT&L  Activity 

1.  Provide  updates  to  system 
documentation;  Review  system  for 

HSI  input.  Program  Management, 
Logistics,  Testing,  Interoperability 
system  evaluation  processes  using 
DAPS  methods. 

Shattuck,  2009;  DoDD 
5000.1 

Operational  Test  Readiness 
Review  (OTRR),  the 
Assessment  of  Operational 
Test  Readiness  (AOTR),  and 
the  Physical  Configuration 
Audit  (PCA) 

2.  Verify  Warfighter  requirements; 

3.  Update  PSP  HSI  criteria 
(requirements  and  performance 
metrics  use  to  optimize  the  system) 
within  the  document. 

Shattuck,  2009 

Product  Support  Plans  (PSP) 
and  Pre-Initial  Operational 
Capability  (IOC)  review 

4.  HSI  (as  part  of  IPTs)  help  to  identify 
testable  HSI  metrics. 

5.  OT&E:  Ensure  that  suitability  and 
effectiveness  meets  the  requirements 

6.  LFT&E:  Lessons  learned  from 
system  design  affects  the 
susceptibility  and  recoverability  of  the 
system. 

Shattuck,  2009 

Initial  Operational  Test  and 
Evaluation  (lOT&E)  and  the 
Live  Fire  Test  and  Evaluation 
(LFT&E) 

7.  HSI  criteria  updates. 

Shattuck,  2009 

Acquisition  Strategy, 
Acquisition  Plan,  Source 
Selection  Plan,  RFPs  and 

Final  Product  Baseline; 
manpower  estimations, 
economic  analyses.  Critical 
Safety  List  items,  PESHE 
and  TEMP 
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e.  Operation  and  Support  Phase 

Life-cycle  sustainment  planning  and  execution  seamlessly  span  a 
system's  entire  life  cycle,  from  Materiel  Solution  Analysis  to 
disposal.  It  translates  force  provider  capability  and  performance 
requirements  into  tailored  product  support  to  achieve  specified  and 
evolving  life-cycle  product  support  availability,  reliability,  and 
affordability  parameters.  (DAU  Online,  2010) 

The  purpose  of  the  Operations  and  Support  (O&S)  phase  is  to 
“execute  a  support  program  that  meets  materiel  readiness  and  operational 
support  performance  requirements  and  sustains  the  system  in  the  most  cost- 
effective  manner  over  its  total  life  cycle”  (DAU  Online,  2010).  The  O&S  phase 
fulfills  this  purpose  with  two  efforts:  Life  Cycle  Sustainment  and  Disposal.  HSI 
practitioners  assist  in  supportability  assessments,  conduct  modeling  and 
simulation  (M&S)  testing,  and  provide  updates  to  documentation.  The 
subsequent  paragraphs  describe  HSI  assistance  during  sustainment  and 
disposal. 

During  sustainment,  the  program  manager  and  logisticians  are 
concerned  with  the  operational  readiness  and  safety  of  the  system.  To  maintain 
the  readiness  and  safety  of  the  system,  supportability  assessments  and  M&S 
activities  take  place  to  determine  possible  system  improvements.  HSI 
practitioners  can  assist  in  these  processes  by  contributing  to  the  supportability 
analysis,  and  supporting  modeling  and  simulation  studies.  During  these  M&S 
studies,  sustainability  and  mission  effectiveness  can  be  assessed.  Lessons 
learned  during  operational  field-testing  are  used  for  modifying  subsequent 
system  increments.  Because  of  these  changes,  each  increment  requires  an 
update  to  the  ODD,  CPD,  and  Key  Performance  Parameter  (KPP)  documents. 

Follow-On  Test  and  Evaluation  is  performed  when  a  system  (or 
product)  is  tested  in  an  operational  environment.  During  this  phase  of  testing, 
verification  of  the  following  suitability  effectiveness  aspects  occurs: 
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Availability,  compatibility,  transportability,  interoperability,  reliability, 
wartime  usage  rates,  maintainability,  safety,  human  factors, 
manpower  supportability,  logistics  supportability,  and  training 
requirements,  environmental  conditions  of  use  scenarios  (i.e., 
weather  and  climatic  conditions,  terrain  effects,  battlefield 
disturbances,  and  enemy  threat  conditions),  mission  length 
performance,  vulnerability  to  countermeasures,  and  susceptibility  of 
a  system.  (Shattuck,  2009) 

Disposal  is  the  final  process  of  any  defense  acquisition  program. 
Disposal  costs  are  a  small  portion  of  the  system’s  total  life  cycle  cost;  however, 
cost  decisions  occur  prior  to  Milestone  B  (See  Figure  8).  The  Defense  Acquisition 
Guidebook  (2010)  states  that  program  managers  must  determine  the  cost 
associated  with  “identification  of  plans  for  minimization  and/or  a  safe  disposal  of 
hazardous  materials,  wastes,  and  pollutants  associated  with  the  system,  and  a 
compliance  schedule  for  National  Environmental  Policy  Act  (NEPA)”  (DAU,  2010, 
p.  283).  Although  indirectly  associated,  HSI  practitioners  (i.e..  Environmental, 
Safety  and  Occupational  Health)  provide  input  to  help  determine  these  costs.  HSI 
practitioners  ensure  that  environmental  precautions  (i.e.,  hazardous  material 
exposure,  safety  measures  and  risk  mitigation)  that  will  affect  the  user  are 
included  in  programs. 
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Figure  8.  Weapon  System  Timeline  and  Life  Cycle  Cost  Categories  (From 

Mislick,  2009) 


Finally,  the  technical  review  process  that  occurs  during  the  O&S 
phase  is  the  In-Service  Review  (ISR).  The  ISR  “ensures  that  the  system  is 
operationally  employed;  system  risk  is  well-understood,  properly  managed,  and 
designed  to  describe  and  characterize  the  health  of  the  system  and  provides 
assessments  that  are  measureable  for  risk,  readiness,  technical  status  and, 
trends”  (Shattuck,  2009).  Although  the  Systems  Safety  Working  Group  and  the 
Integrated  Logistics  Management  Team  conduct  the  ISR,  it  must  receive 
contributions  from  HSI  practitioners  to  ensure  the  system  is  performing  as 
specified.  Table  6  summarizes  the  FISI  activities  during  the  O&S  phase. 
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Table  6.  O&S  Phase  HSI  Activity  Summary 


HSI  Activity 

Publication 

IDAT&L  Activity 

1 .  Maintain  the  readiness  and  safety  of  the 
system,  supportability  assessments  and  M&S 
activities  take  piace  to  determine  possibie 
system  improvements; 

2.  Assess  sustainabiiity  and  mission 
effectiveness. 

DAG,  2010, 

Shattuck,  2009 

Sustainment  supportability 

3.  Assist  in  providing  updates  due  to  system 
increment  improvements. 

DAG,  2010, 

CDD,  CPD,  and  Key 
Performance  Parameter 
(KPP)  documents 

4.  Assist  in  verification  of  the  following 
suitability  effectiveness 

DAG,  2010, 

Follow-On  Test  and 
Evaluation 

5.  HSI  practitioners  (i.e.,  Environmental, 

Safety  and  Occupational  Health)  provide 
input  for  environmental  precautions. 

DAG,  2010, 

Disposal 

6.  HSI  practitioners  provide  input  to  ensure 
the  system  is  performing  as  specified. 

Shattuck,  2009 

In-Service  Review  (ISR). 

Sections  (a)  through  (e)  have  identified  HSI  activities  from  Concept 
identification  to  Disposal.  It  is  now  possible  to  create  a  more  accurate  picture  of 
where  HSI  activity  costs  can  occur  as  well  as  possible  drivers  that  can  contribute 
to  the  cost  estimation  processes.  The  next  section  will  provide  an  overview  of  the 
cost  estimation  techniques,  methods  and  processes  used  to  develop  a  new  HSI 
cost  estimation  process. 

B.  COST  ESTIMATION  TECHNIQUES,  METHODS  AND  PROCESSES 

Cost  estimating,  as  part  of  a  total  systems  analysis,  provides  an 
analytic  underpinning  to  support  decision  makers.  (Nussbaum, 

2010) 

This  section  discusses  the  basic  steps  of  cost  estimation.  The  topics 
include  Cost  estimation  purpose,  scope,  ground  rules  and  assumptions; 
Modeling  and  identification  of  cost  drivers,  and  cost  estimation  methodology. 
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1.  Purpose,  Scope,  Ground  Rules  and  Assumptions 


In  any  cost  estimation  activity,  there  are  several  issues  that  need  to  be 
determined  at  the  beginning  of  the  process.  These  issues  include  the  purpose, 
scope,  ground  rules,  and  assumptions  for  the  cost  estimation  process  about  to 
be  undertaken.  In  keeping  with  this  approach,  this  section  makes  explicit  each  of 
these  issues. 


a.  Purpose 

The  purpose  of  the  present  effort  is  to  refine  current  cost  estimation 
processes  that  include  HSI  activities  that  take  place  during  the  entire  acquisition 
life  cycle.  The  refined  processes  will  be  used  during  the  AoA  as  well  as  during 
other  activities  such  as  the  program  office  estimate  and  the  baseline  cost 
estimates. 


b.  Scope 

The  cost  estimation  process  will  attempt  to  address  all  HSI 
activities  throughout  the  entire  acquisition  life  cycle.  The  process  is  intended  to 
be  generic  in  nature  so  that  it  can  be  applied  to  any  military  service  and  to  any 
type  of  system.  However,  the  research  effort  will  emphasize  issues  such  as 
development/operation  test  and  evaluation  processes,  program  increments, 
operation  and  support,  and  environmental  costs.  The  cost  estimation  process 
should  also  include  risk  and  cost  relationships. 

c.  Ground  Rules 

The  present  effort  is  constrained  by  manpower  (this  thesis  is  an 
individual  effort  by  one  graduate  student),  by  time  (this  thesis  had  to  be 
completed  within  nine  months)  and  by  availability  of  other  resources  such  as  cost 
data  and  guidance,  subject-matter  experts,  and  funding  for  travel. 
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d.  Assumptions 


The  work  reported  herein  is  only  an  initial  effort;  additional  work  will 
be  required  to  refine  the  cost  estimation  process  for  HSI  activities.  The  process  is 
based  upon  the  data  that  were  collected  from  subject-matter  experts  who  were 
available.  Although  the  current  research  is  based  predominantly  on  peacetime 
operations,  future  efforts  should  concern  war-related  cost. 

2.  Identification  and  Modeiing  of  Cost  Drivers 

The  life-cycle  phase  review  described  earlier  identified  HSI  activities  that 
are  essential  to  the  AoA  cost  estimate  process.  In  performing  cost  estimates,  HSI 
driver  trade-offs  must  also  be  determined.  These  estimates  include  the  activities 
that  help  determine  the  projected  cost  of  all  HSI  domains.  However,  emphasis  is 
placed  on  manpower,  training,  personnel,  and  human  factors  engineering.  This 
section  discusses  these  cost  drivers,  cost  estimate  relationships,  and  modeling 
efforts. 


a.  Manpower  Cost  Drivers 

Manpower  cost  drivers  and  relationships  are  the  most  commonly 
used  and  accepted  variables  in  cost  estimate  methodology.  The  publications  that 
discuss  Manpower  cost  drivers  and  relationships  include  the  Directive-Type 
Memorandum  (DTM)  09-007  and  the  2010  Federal  Acquisition  Regulations. 

The  DTM-09-007  states  that  manpower  costs  are  identified  as 
“specific  unit,  organization,  function,  mission,  or  defense  acquisition  program... 
analysts  should  report  the  full  costs  of  both  military  and  civilian  DoD  manpower” 
(DoD,  2010a,  p.  5).  Cost  drivers  such  as  man-hour  costs,  are  incorporated  into 
the  O&S  phase  of  the  life  cycle.  Although  it  is  imperative  to  include  the  proper 
workforce  mixture  (military,  civilian  and  contractor  workloads),  the  highest 
performance  at  minimum  cost  is  needed.  Therefore,  AoA  processes  include 
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estimates  for  comparison  between  these  entities.  These  costs  do  not  usually 
occur  once,  but  span  over  the  life  of  the  system.  For  example,  at  any  given  point 
in  time,  the  cost  of  a  military  member  will  include: 

•  Base  pay, 

•  Basic  substance  and  housing  allowance, 

•  Cost  of  medical  care,  and 

•  Retirement 

The  contractor  cost  may  be  short-lived.  Contractor  costs  usually  include 
development  or  short-term  services  involving  testing,  logistics,  transportation, 
and  will  not  include  added  benefits.  Defining  manpower  business  rules  and 
methodology  is  the  next  step  for  defining  manpower  cost  estimation 
relationships. 

The  DTM-09-007  defines  business  rules  and  methodology  for  the 
workforce  mix  (DoD,  2010a).  In  order  to  determine  appropriate  manpower  costs, 
composite  rates  for  Manpower  must  be  used.  The  Office  of  Undersecretary  of 
Defense  website  provides  these  rates.  Appendix  A  of  this  thesis  also  contains  the 
current  fiscal  year  composite  rates.  Furthermore,  determining  manpower  cost 
includes  describing  direct  and  indirect  costs. 

Direct  costs  that  are  often  overlooked  include  the  deferred 
compensation  costs  (liabilities/retirement  or  pay-as-you-go  cost)  that  DoD 
civilians  may  incur  (p.  7).  Many  tests  and  evaluations  during  development  invoke 
hearing  conservation  programs.  Occupational  Health  costs  incorporate  these 
programs  and  include  hearing  test  costs  or  disability  costs  after  the  fact.  The 
DTM-09-007  states  the  “full  costs  of  manpower  include  current  and  deferred 
compensation  costs  paid  in  cash  and  in-kind,  as  well  as  non-compensation 
costs”  (DoD,  2010a,  p.  7).  Short-term  costs  increase  when  workforce  duration 
lengthens.  These  costs  are  associated  with  activities  such  as  establishing  or 
modifying  day  care  and  commissary  services  for  military  locations.  Table  7 
includes  an  overview  of  the  direct  costs  incurred. 
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Table  7.  Direct  Labor  Cost  Elements  for  Military  and  DoD  Civilian  Personnel  (From 

DTM-09-007:  DoD,  [2010]) 


WHaiY 

Civilian 

DoD 

Other  Federal  Agency 

DoD 

Other  Federal  Agency 

Vanable  Costs  in 
Short  Rut 

Basic  pay 

Alowanc«s  and  spadal  pays 
rtcettv«pa^ 

Heath  beneiH.  acOve  duly  and  dependents 

Social  Secutty  and  Medicare 

Retied  pay (accrtjal) 

T  lavelRCSitranspoitaton  subsidy 

Educat  on  assistance 

Heath  beneft  lebiee  (>d5  MERHCF  accnial) 
Tiainlng  costs  (amcrtiaed  over  years  cl  practee) 
Recnjtmert  advertising,  etc  (amortsdi 

Conorert  receipt  (Treasury) 
Military  Retremert  (Treasury) 
HeRHCFfTreasuy) 
Chtleducaton(Educaeon| 

Basic  payflocality  pay 

Akwarves  and  special  pays 
hcertvftPerfomtance  awards 

Health  benefit  (government  share  of  FEhSP) 

Social  Seoirty  and  Medicare 

Retied  pay  (government  share) 
Tra«IPCSAransportatcn  subsidyieiccabon  borus 
Educabonassistarvte 

Owitmejholldayroiherpays 

Lie  1  nsurance'wcirkei's  conpensaton  benefits 
Recrubng,  advertsmQ.  eC.  (amortized) 

Pied  Costs  in 
Short  Rji 

Child  deveiopmeni 

Famiystiiport  services 

Discotrigrocehes 

Chikl  developmert 

Defetred 

PayAs-YoihGo 

Costs 

Heath  benelil  lebiee  (<8E  letiee  and  family) 
Heath  benelil  other  fTAhff’  and  CHCBP) 
Discoutgrocehes.  retree 

Separaton  pay  and  travel 

Unemploymeit  benefits 

Death  gratjtes 

Stiviva  benefits 

VA  benefits  (Veterans  Affairs) 
Employmeit  training  (Labor) 

Seveience  heath  benefit 

Severance  payfincentve 

Retirement  benefit  (CSRS  iifiiided) 
Heath  benefit 

Life  instance  benefit 

The  2010  Federal  Acquisition  Regulation  section  31.203  defines 
indirect  costs.  Indirect  costs  are  those  costs  that  are  “not  directly  identified  with  a 
single  final  cost  objective,  but  identified  with  two  or  more  final  cost  objectives  or 
with  at  least  one  intermediate  cost  objective”  (Department  of  Defense  [DoD],  U.S. 
General  Services  Administration  [GSA],  National  Aeronautics  and  Space 
Administration  [NASA],  2010).  Direct  costs  can  be  treated  as  indirect  costs  if  the 
dollar  amount  is  minor  and  the  accounting  treatment  is  applied  consistently. 
These  cost  must  however,  produce  results  that  are  the  same  as  if  treated  as  a 
direct  cost  (DoD  et  al.,  2010).  These  costs  include  machinery,  facilities,  materials 
and  supplies;  however,  they  vary  from  contract  to  contract. 

Frequently,  DoD  organizations  write  contracts  for  FISI  services; 
therefore,  evaluating  indirect  cost  is  essential.  Unlike  the  work  completed  by  DoD 
or  civilians,  many  of  the  liabilities  for  contracting  are  not  under  sovereign 

immunity  and  must  be  factored  in  to  the  cost  equation.  If  contract  negotiations 
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leave  out  goods,  benefits,  and  services  for  government-furnished  property  and 
cost  of  upper-level  management,  contracts  will  most  likely  increase  in  cost  over 
time. 

The  DTM-09-007  states  that  these  overlooked  contract  costs  can 

include: 

•  Liability  to  third  parties; 

•  Reimbursements  to  a  contractor  for  payments  the  contractor,  its 
insurance  company,  or  the  Department  of  Labor  make  pursuant  to 
Public  Laws  85-608  and  77-784  (References  (i)  and  (j)); 

•  Reimbursements  the  Department  of  Defense  is  obligated  to  make 
with  respect  to  publicly  sponsored  insurance  (e.g.,  air  carrier 
insurance  sponsored  by  the  Federal  Aviation  Administration 
pursuant  to  title  49,  U.S.C.  (Reference  (k))  and;..., 

•  The  value  of  the  authorization  and  consent  to  infringe  privately  held 
patents  with  freedom  from  monetary  damages  and  injunctive  relief 
that  would  otherwise  prevent  a  contractor  from  performing  the 
function.  (DoD,  2010,  p.  12) 

As  with  any  system,  analyzing  the  operational  tempo  and  mission 
requirements  will  provide  other  influential  cost  drivers.  Johnson,  Osborn,  Previc, 
and  Prevost  (2005)  paper  provides  an  excellent  example  of  influential  cost 
drivers. 

The  authors  addressed  drivers  by  performing  a  top  down 
requirements  analysis  (TDRA).  This  approach  determined  that  the  highest 
manning  occurred  during  operational  tempos  under  battle  readiness  and  wartime 
amphibious  assault  operations.  These  conditions  lead  to  increased  watch 
standing  and  manning  requirements.  By  including  these  types  of  influential 
drivers  in  the  estimate,  the  impact  of  such  outcomes  as  the  need  for  equipment 
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redesign,  lower  than  expected  performance,  and  delays  in  schedule  can  be 
determined.  These  types  of  drivers  are  essential  in  producing  reliable  estimates 
for  HSI  related  work. 

b.  Personnel  and  Training  Cost  Drivers 

Personnel  and  Training  domains  also  provide  important  cost  drivers 
and  relationships.  Each  grade,  rank,  rate,  general  service  position  or  manpower 
contract  type  will  have  specific  associated  costs  and  are;  therefore,  factored  into 
the  estimate  for  O&S  phase.  Determining  the  personnel  requirements  and 
recruiting  efforts  for  supplying  the  needed  operators  and  maintainors  of  the 
system  is  another  personnel  cost. 

The  training  costs  involved  include  developing  and  updating 
training  strategy  and  plans,  as  well  as  developing  effective  training  methods  and 
tools.  To  produce  qualified  maintainors  and  operators  for  the  various  training 
pipelines,  these  methods  and  tools  are  essential.  Training  drivers  include  the 
technologies  used  as  well  as  the  test  and  evaluation  of  technology.  Drivers 
should  also  consider  the  manpower  cost  to  deliver  the  instruction.  Therefore, 
both  the  R&D  and  O&S  life-cycle  phases  consider  training  costs. 

c.  Human  Factors  Engineering  (HFE)  Cost  Drivers 

The  HFE  domain  dominates  Research  and  Development  (R&D) 
cost  estimates.  The  HFE  cost  drivers  in  R&D  are  essentially  the  cost  of 
development,  testing  and  evaluation.  Developing  tools,  analyzing  tasks  and 
evaluating  human-technology  interactions  are  typical  activities  for  HFE 
practitioners.  These  activities  can  account  for  hundreds  and  even  thousands  of 
man-hours  contribute  to  these  methods  and  research  processes  and  thus  should 
qualify  as  a  high  driver. 

Modeling  and  Simulation  (M&S)  costs  occur  in  practically  every  HSI 
domain  activity  (i.e.,  manpower,  training,  safety  and  survivability)  in  a  acquisition 
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program.  For  example,  HFE  costs  drivers  that  relate  to  manpower  include 
function  allocation,  workforce  role  determination,  workload  reduction,  as  well  as 
the  inherent  workload  risk  and  affordability  determinations.  Therefore,  M&S  is  a 
high  driver  depending  on  the  cost  of  technology,  length  of  service,  and  the  needs 
determination  for  modification  of  design.  Use  of  analogous  systems  and 
comparable  tools  help  determine  the  cost  estimations. 

d.  Environment  and  Safety  Cost  Drivers 

As  stated  previously.  Environment  and  Safety  have  their  own 
process  owners.  Periodically,  practitioners  must  review  and  update  safety  and 
environmental  inspection  procedures,  design  requirements,  and  regulations. 
Besides  licensure  and  environmental  fees,  a  majority  of  safety  costs  are 
comprised  of  man  hours  required  to  perform  needed  administration  and 
procedural  requirements.  The  Environmental,  Safety  and  Health  Cost  Analysis 
guide  (1998)  define  two  drivers  of  environmental  cost: 

•  Public  Law  103-337,  Section  815,  Environmental  Consequence 
Analysis  of  Major  Defense  Acquisition  Programs,  and 

•  Department  of  Defense  Regulation  5000.2-R,  Mandatory 
Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and 
Major  Automated  information  System  (MAIS)  Acquisition  Programs, 
Section  4.3.7.  (p.  9) 

These  programs  discuss  “materials  to  be  used,  the  mode  of  operations  and 
maintenance,  requirements  for  demilitarization,  and  methods  of  disposal,  after 
consideration  of  all  pollution  prevention  opportunities  and  in  light  of  all 
environmental  mitigation  measures  to  which  the  Department  expressly  commits” 
(Air  Force  Materiels  Command,  1998). 

The  Environment  and  Safety  practitioners  work  hand  in  hand  with 
HFE  practitioners  when  designing  the  system  for  safe  and  effective  handling, 
operation,  and  disposal.  These  are  mainly  manpower  intensive  drivers;  however 
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when  a  system  has  a  lifespan  that  exceeds  initial  life  expectancy,  the  cost  of 
reinstallation,  deployment,  upgrades  and  disposal  will  skyrocket.  An  example  of  a 
system  that  has  gone  beyond  the  normal  lifespan  and  has  required  significant 
modification  is  the  Iowa  class  battleships.  Commissioned  in  1942,  the  Iowa  class 
ships  received  numerous  modifications.  Defense  and  survivability  modifications 
included  electronic  warfare  suites,  close-in  weapon  systems,  BGM-109 
Tomahawk  Land  Attack  Missiles,  and  RGM-84  Harpoon  anti-ship  missiles.  After 
48  years  of  service,  the  ships  were  finally  decommissioned. 

e.  Occupational  Health,  Habitability  and  Survivability  Cost 
Drivers 

The  final  HSI  domains  of  Occupational  Health,  Habitability  and 
Survivability  are  man-hour  intensive.  These  domains  contribute  cost  drivers 
indirectly  to  the  cost  estimation.  They  provide  cost  variables  for  the  R&D  phase 
of  the  life  cycle.  Bratt,  Doganiero,  and  Spencer  (1998)  discuses  these  human 
error  and  mishap  related  variables. 

As  stated  previously,  indirect  manpower  costs  include  deferred 
compensation  (e.g.  disability).  Bratt,  et  al.  (1998)  suggests  that  it  is  important  to 
substantiate  the  driver  causal  factors;  otherwise,  consequences  may  be 
undetermined  for  any  system  development.  Mishap  data  is  relevant  to  any 
platform  whether  it  is  aircraft,  ship  or  land  based  (including  peace  and  war 
operations).  The  leading  causes  of  mishaps  are  human  error  and  equipment 
failure. 

Human  error  may  occur  from  someone  having  a  bad  day,  a  crisis  in 
their  personal  life,  delayed  or  incorrect  decision-making,  or  even  unfamiliar  or 
faulty  written  procedures.  In  many  cases,  the  equipment  failure  also  stems  from 
human  error  (e.g.  errors  in  design  considerations  etc.).  Manpower  reduction, 
poor  or  inadequate  human  factors  engineering,  and  lack  of  attention  to  safety 
considerations  are  among  the  factors  that  contribute  to  these  efforts.  The 
potentially  enormous  price  incurred  for  these  errors  translates  into  cost  such  as: 
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•  disposal  costs, 

•  medical  care, 

•  disability  and  rehabilitation, 

•  redesign  and  modification,  and 

•  training 

Identifying  and  mitigating  the  health  risks  involved  in  system  development  is  a 
cost  avoidance  measure.  Although  not  normally  considered  in  a  cost  estimate, 
these  potential  outcomes  certainly  should  be  included.  Human  error  is 
unavoidable,  though  it  can  be  reduced.  Cost  estimates  must  include  mitigation  of 
these  risks  and  mishaps  (apparent  in  any  system)  and  be  compensated  for  when 
they  occur. 


f.  Cost  Modeling 

HSI  cost  modeling  can  be  adapted  from  current  models  in 
Occupational  Health  and  System  Engineering.  These  modeling  activities  are 
dependent  on  the  system  and  target  audience  description  for  the  intended  use  of 
the  system.  This  section  discusses  these  various  models  used  in  cost  estimation 
of  Occupational  Health  and  Systems  Engineering. 

Bratt  et  al.  (1998)  reports  on  a  1997  Army  weapon  hazard  study 
that  linked  possible  health  hazards  to  clinic  services.  Specifically,  the  authors 
summarized  their  work  as  follows: 

Incidence  rates  and  calculated  costs  based  on  industry-wide  data 
on  injuries,  lost  time,  hospitalization,  and  disability,  and  this 
framework  provides  a  method  to  reasonably  estimate  the  medical 
and  lost  military  manpower  costs  of  unabated  health  hazards 
associated  with  Army  materiel,  (p.  443) 

The  article  addressed  the  effectiveness  of  the  health  risk  assessment  and 
management  system  while  also  contributing  to  the  control  of  life  cycle  cost.  The 
article  provides  the  severity  levels  and  hazard  probabilities  needed  for  cost 
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estimations  (see  Figure  9,  Tables  8  and  9).  The  resource  also  includes  valuable 
equations  for  factoring  hazard  variables  into  the  cost  estimations.  A  full  variable 
list  and  explanations  are  available  in  Appendix  B. 


Total  Medical  Costs 


Figure  9.  Cost  Components  from  a  Single  Hazard  (From  “Estimating  the 

health  hazard  cost,”  1998) 


Table  8.  Risk  Assessment  Codes  (RAC)  and  Costs  (Thousands  of  Dollars)  Matrix 

(from  “Estimating  the  health  hazard  cost,”  1998) 


Hazard  Severity  (SJ 

Hazard  Probability  (PJ 

Frequent 

Probable 

Occasional 

Remote 

Improbable 

A  (.9) 

B(.5) 

C  (.2) 

D  (.01) 

E  (.001) 

I  (1) 

Catastrophic 

1  (Si  5,088) 

1  (S8,471) 

1  ($3,508) 

2  ($365) 

3  ($216) 

II  (.1) 

Critical 

1  (SI  ,410) 

1  ($783) 

2  ($313) 

3  ($16) 

4($1) 

III  (.01) 

Marginal 

2  ($137) 

3  ($76) 

3  ($30) 

4  ($2) 

5  ($0,152) 

IV  (.001) 

Negligible 

3  ($13) 

5  ($7) 

5  ($3) 

5  ($0,148) 

5  ($0,015) 

Notes:  The  calculations  are  based  on  a  high  risk  system.  The  numbers  1,  2,  3,  4,  and  5,  in  the  columns  under 
Hazard  Probability  are  the  RACs.  The  numbers  in  parentheses  in  the  columns  under  Hazard  Probability  are 
the  medical  costs  that  are  incurred  for  a  given  RAC  if  no  intervention  occurs. 
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Table  9.  Cost  Component  Equations  (From  “Estimating  the  health  hazard  cost,” 

1998) 


Re4ate<l  Co«t 
Component 

Component 

calctiiation 

E<|uation 

All  (except 
<ieath  costs) 

Number  of  people 
exposed  to  hazard 

N.  =  P.xN,x 

Clinic  costs 

Number  of  people 
injured  or  91 

N.  =  N.  X  S.  X  1. 

Clinic  costs 

Clinic  costs 

C.  =  N.  X  F. 

Clinic  costs 

Number  of  dinic  visits 

N.  =  N.  X  S,.  X  [V.  (1,  X  N  J] 

H  ospital  iza  tion 
costs 

Hospitalization  costs 

=  N„  X  F„ 

H  ospital  tzalion 
costs 

Number  of  persons 
hospitalized 

=  N.  X  S,  X  1, 

H  ospital  iza  tion 
costs 

Number  of  hospital  days 

N,  =  N.  X  S.X  l,x  S  (D«  X 

Lost  time  costs 

Lost  time  costs 

C,  =  N,  X  X  B, 

Lost  time  costs 

Number  of  persons 
losing  time 

Np,  =  N.  X  S,  X  1, 

Lost  time  costs 

Number  of  lost 
workdays 

N,  =  N.  xS,xl„x5:(D„x 

Qisabittty  costs 

Disabilfty  costs 

X  X  1,  X  X  X  (D^  X  X  12  mo/yr 

+  [{I,  X  B,)  +  (Ip  X  Bp)]  ’ 

I>tsabifrty  costs 

Number  of  persons 
disabled 

=  N.  X  S,  X  rr.  X  I,  +  I,  *  Ip) 

RetiaPiiitabon 

costs 

Rehabilitation  costs 

C,  =  N.  X  S„  X  l.x  T,  X  X  D,  X  Q,x  B^ 

ReHabilftation 

costs 

Number  of  rehabilitation 

cosoc 

N,  =  N.  X  X  I,  X  T,  X  XD,  X  Q, 

Deatti  costs 

Death  costs 

X  B^ 

Mote:  Number  ot  people  exposed  to  hazard  (N,  -  P«x  Ns  x  N^  is  a  common  term  reisted  to  an  cost  compo¬ 
nents  except  death  costs. 

Since  the  System  Engineering  community  undoubtedly  influences 
the  HSI  community,  Valerdi  (2005)  provides  further  development  of  possible  cost 
estimation  relationships  that  are  fundamental  to  HSI-derived  models. 

Valerdi  identifies  system  engineering  effort  (in  person-months) 
using  the  Constructive  Systems  Engineering  Cost  Model  (COSYSMO).  This 
model  includes  four  primary  drivers  (see  Table  9),  as  well  as  fourteen  effort 
multipliers/cost  drivers  (See  Table  10). 
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Table  10.  Size  Drivers  and  Corresponding  Data  Items  (From  “The  Constructive 

Systems  Engineering  Cost  Model,”  2005) 


Dii\er  Name 

Data  Item 

#  of  System  Requirements 

Counted  fiom  the  system  specification 

#  of  Major  Interfaces 

Coimted  fiom  interface  control  document(s) 

#  of  Critical  Alaoritlmis 

Coimted  fiom  system  spec  or  mode  description  docs 

#  of  Operational  Scenarios 

Coimted  fiom  test  cases  or  use  cases 

Table  1 1 .  Cost  Drivers  and  Corresponding  Data  Items  (from  “The  Constructive 

Systems  Engineering  Cost  Model,”  2005) 


Driver  Name 

Data  Item 

Requirements  understandmg 

Subjective  assessment  of  tlie  ss’stem  requirements 

Architecture  understanding 

Subjective  assessment  of  the  system  architecture 

Level  of  service  requirements 

Subjective  difficulty  of  satisfymg  the  key  performance 
parameters 

Migration  complexity 

Influence  of  legacy  system  (if  applicable) 

Technology  risk 

Matunty.  readmess.  and  obsolescence  of  technology 

Documentation  to  match  hfe  cycle 
needs 

Breadth  and  depth  of  required  documentation 

#  and  Diversitv'  of 

Sites,  mstallations.  operatmg  environment,  and  diverse 

mstallations/platforms 

platforms 

#  of  Recursive  levels  m  the  design 

Number  of  apphcable  levels  of  the  Work  Breakdown  Structure 

Stakeholder  team  cohesion 

Sub]ective  assessment  of  all  stakeholders 

Personnel'team  capabilifv 

Subjective  assessment  of  the  team’s  mtellectual  capabihts' 

Personnel  experience/continuitv 

Subjective  assessment  of  staff  consistency 

Process  capabihU' 

CMMI  level  or  equivalent  rating 

Multisite  coordmation 

Location  of  stakeholders  and  coordmation  barriers 

Tool  support 

Subjective  assessment  of  SE  tools 

Valerdi  initially  expresses  System  Engineering  effort  in  terms  of 
historical  data,  size  drivers  and  effort  multipliers.  The  final  derived  formula  for 
system  engineering  effort,  however,  includes  weighted  variables  for  the  effort 
multipliers  (See  Figures  10  and  11).  These  equations  identify  the  relationships 
between  the  effort  multipliers  and  the  cost  factor  variables.  Each  factor  is  subject 
to  the  changes  in  system  use  such  as  “redesign,  reimplementation,  and  retest” 
(Valerdi,  2005,  p.  41).  Factors  are  also  impacted  by  complexity  and  volatility.  The 
model  encompasses  periodic  adjustment  for  the  drivers.  Furthermore,  Valerdi 
states  that  increases  or  decreases  to  variables  are  system  or  criteria  dependant. 
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n 


Equation  2 


PM ^  =  -4  ■  iSizey  -  n  EM, 

7=1 


W'liere: 

P>I>.-s  =  et'fbiT  ill  Person  XIontlis  (IS'oniinal  Scliednle) 

A.  =  calibration  constant  derived  from  liistorical  project  data 

Size  =  determined  by  compntina  tlie  weighted  sum  of  the  fom  size  dii\'ers 

E  =  represents  economy/dis economy  of  scale;  default  is  1.0 

u  =  number  of  cost  di  ivers  Cl-^) 

E!\I,-  =  effoit  multiplier  for  the  /tb  cost  driver.  Nominal  is  1.0.  Adjacent 
multipliers  ha\"e  constant  ratios  (geometiic  progression).  Wtithin  their 
respecti\'e  rating  scale,  the  calibrated  sensitivity  range  of  a  mnlfiplier  is  the 
ratio  of  highest  to  lowest  value. 


Figure  10.  COSYSMO  Operational  Form  (From  “The  Constructive  Systems 

Engineering  Cost  Model,”  2005) 


Equation  6 


Figure  1 1 .  Final  COSYSMO  CER  (From  “The  Constructive  Systems 

Engineering  Cost  Model,”  2005) 

In  Figure  11,  the  symbol  expressions  include; 


k  =  number  of  requirements,  interfaces,  algorithms,  and  operational 
scenarios  (REC,  INTF,  ALG,  OPSC) 
w  =  weight 
e  =  easy 
n  =  nominal 
d  =  difficult 
O  =  driver  count 
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Next,  Valerdi  establishes  categories  for  the  COSYSMO  model 
factors  (See  Figure  12).  In  HSI,  identity  of  such  factors  is  necessary  due  to  the 
role  of  the  HSI  practitioner  in  policy  writing,  development,  testing,  modeling  and 
simulation  of  systems. 


UNDERSTANDING  FACTORS 

-  Requirements  understanding 

-  Architecture  understanding 

-  Stakeholder  team  cohesion 

-  Personnel  experience/continuity 

COMPLEXITY  FACTORS 


PEOPLE  FACTORS 

-  Personnel/team  capability 

-  Process  capability 

ENVIRONMENT  FACTORS 

-  Multisite  coordination 

-  Tool  support 


-  Level  of  service  requirements 

-  Technology  Risk 

-  #  of  Recursive  Levels  in  the  Design 

-  Documentation  Match  to  Life  Cycle  Needs 


OPERATIONS  FACTORS 

-  #  and  Diversity  of  Installations/Platforms 

-  Migration  complexity 


Figure  12.  Cost  Factor  Clustering  (From  “The  Constructive  Systems 

Engineering  Cost  Model,”  2005) 


Finally,  Figure  13  and  Table  12  depict  Valerdi’s  final  cost  model 
(and  the  acronym  listing).  The  COSYSMO  model  provides  the  foundation  that 
HSI  models  can  build  upon.  Future  HSI  modifications  should  enhance  this 
current  model. 


Equation  14- 

log[SH_HRS_-A.DJ]  =  loafSIZE]  -t-  loaCETNTDHRSTANDrNrG]  — 

log  [COMPLEXITY]  —  log[OPER.A.TI01srS]  — 
logfPEOPLE]  H-  logfElsATROKTlVIENT] 

LTSTDERSTANDING  =  REQLT  *  ARCH  ♦  TEAM  *  PEXP 
COMPLEXITY  =  LSVC  *  TRSKl  *  RECLT  *  DOCLT 
OPERATIONS  =  INST  *  MIGR 
PEOPLE  =  PCAP  *  PROC 
EN^^RONMENT  =  SITE  *  TOOL 


Figure  13.  COSYSMO  Model  (From  “The  Constructive  Systems  Engineering 

Cost  Model,”  2005) 
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Table  12.  COSYSMO  Predictor  Descriptions  (From  “The  Constructive  Systems 

Engineering  Cost  Model,”  2005) 


Predictor 

Term 

Description 

s, 

log(REQ) 

#  of  System  Requirements 

Sj 

log(INTF) 

#  of  Major  Interfaces 

S3 

log(ALG) 

#  of  Critical  Algorithms 

S3 

log(OPSC) 

#  of  Operational  Scenarios 

EM, 

log(RQMT) 

Requirements  Understanding 

EMj 

log(ARCH) 

Architecture  Understanding 

EMj 

log(LSVC) 

Level  of  Service  Requirements 

EM4 

log(MIGR) 

Migration  Complexity 

EMj 

log(TRSK) 

Technology  Risk 

EMe 

log(DOCU) 

Documentation  to  match  lifecycle  needs 

EMt 

log(INST) 

#  and  diversity  of  installations/platforms 

EMe 

log(RECU) 

#  of  recursive  levels  in  the  design 

EM9 

log(TEAM) 

Stakeholder  Team  Cohesion 

EM,o 

log(PCAP) 

Personnel/Team  Capability 

EM„ 

log(PEXP) 

Personnel  Experience/Continuity 

EM, 2 

log(PROC) 

Process  Capability 

EM, 3 

log(SITE) 

Multisite  Coordination 

EM,3 

log  (TOOL) 

Tool  Support 

3.  Cost  Estimation  Methodoiogy 

Cost  estimators  typically  use  four  methodologies.  Each  methodology  is 
suited  for  different  periods  in  the  life  cycle  although  they  may  be  used  in  any  of 
the  life  cycle  periods.  These  methods  are;  engineering  build-up  (bottom-up 
technique);  analogy;  parametric;  and  expert  opinion.  This  section  discusses 
these  methods  and  describe  the  manner  in  which  they  are  used. 

Acquisition  programs  frequently  employ  a  Work  Breakdown  Structure 
(WBS)  or  a  Cost  Analysis  Requirement  Descriptions  (CARD).  When  a  detailed 
WBS  (at  the  lowest  level)  is  available  for  the  system  of  interest,  the  engineering 
build-up  method  is  the  preferred  cost  estimation  method.  When  other  programs 
of  a  similar  nature  (complexity,  technical  or  physical)  are  available,  the  analogy 
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method  is  preferred.  In  general,  whenever  few  actual  costs  are  available  early  a 
program,  similar  programs  are  the  most  viable  means  for  determining  cost 
estimates. 

The  parametric  method  is  preferred  if  the  program  is  in  the  early  stages  of 
development  and  limited  data  (program  and  technical)  are  available.  This  method 
is  similar  to  the  analogy  method.  It  uses  similar  system  performance  and  design 
characteristics  while  determining  cost  element  relationships  (CER)  on  which  to 
determine  statistical  inference.  If  these  methods  are  not  available,  then  it  is 
possible  to  use  expert  opinion.  It  is  also  possible  to  use  a  combination  of  these 
methods,  if  the  system  allows.  Reproducibility  and  sensitivity  analysis  are 
techniques  used  to  improve  the  accuracy  of  the  cost  estimate.  Figure  14  provides 
a  comparison  of  these  methods’  advantages  and  disadvantages 


Model 

Category 

Description 

Advantages 

1  .imitations 

Analogj" 

Compare  project 
with  past  sirmlar 
projects 

Estimates  are  based 
on  actual 
experience 

Tmly  similar 
projects  must  exist 

Expert 

Judgment 

Consult  with  one  or 
more  experts 

Little  or  no 
historical  data  is 
needed:  good  for 
new  or  unique 
projects 

Experts  tend  to  be 
biased:  knowledge 
level  is  sometimes 
questionable 

Bottoms-E'p 

IndiMduals  assess 
each  component 
and  then 
component 
estunates  are 
summed  to 
calculate  the  total 
estimate 

Accurate  estimates 
are  possible 
because  of  detailed 
basis  of  estimate 
(BOE),  promotes 
indisidual 
responsibihry- 

Methods  are  time- 
consummg;  detailed 
data  may  not  be 
available, 

especially  early  in  a 
program' 

integration  costs  are 

sometimes 

disregarded 

Parametric 

Models 

Perform  overall 
esomate  using 
design  parameters 
and  mathematical 
algorithms 

Models  are  usually 
fast  and  easy  to  use. 
and  useful  early  m 
a  program;  they  are 
also  objective  and 
repeatable 

Models  can  be 
inaccurate  if  not 
properly  calibrated 
and  validated;  it  is 
possible  that 
historical  data  used 
for  cahbration  may 
not  be  relevant  to 
new  programs 

Figure  14.  Methodology  Comparison  (From  Nussbaum,  2010) 
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Figure  15  provides  recommendations  for  which  method  to  use  during 
various  phases  of  the  acquisition  life  cycle. 


Design 

Maturitv 


Program  - > 

Life  Cycle 


Estimating  _ 
Methodologr* 


Figure  15.  The  Cost  Estimating  Process:  Estimating  Approaches  and  Cost 
Methodologies  and  Milestones  (From  Nussbaum,  2010) 


FeasUM&ty  Jnfiftifilfimary  l^taiilJesign 


Requirements  Design  Fab /Integration  Production 

A  A  A 


Concept 

Refinement 

Technology 

Development 

Sys.  Development 
&  Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

Pre-Svstems  Acquisition 

Systems  Acquisition 

Sustainment 

Parametnc**-th££e  is  a  relationship 
bet\('een  cost  and  a  technical  charactenstic, 
such  as  S '  ton 

Actual  Cost  —HTOst  returns 
on  same  or  similar  system 

An  alogy — system  or 
subsystem  is  like  something 
else  for'^hich  the  cost  is 

kno^*n 

Engineering  Bottoms-Dp  - 
build  up  of  costs  fox  smallei, 

disacte  elements  of  the  total 

system 

A  majority  of  MSI  activities  have  not  been  defined  in  most  system  Work 
Breakdown  Structures.  Therefore,  this  thesis  uses  expert  opinion  to  determine 
MSI  costs  during  the  acquisition  life  cycle.  However,  the  long-term  goal  is  to  gain 
enough  HSI  cost  data  in  similar  systems  to  use  the  parametric  method.  This 
method  will  ensure  a  higher  level  of  accuracy  based  on  the  CER  used. 

If  a  parametric  method  can  be  employed,  CERs  are  determined  based  on 
the  following  formula: 

Cost  =  f  (technical,  performance,  schedule)  (Nussbaum,  2010) 

In  doing  so,  it  will  be  possible  to  use  statistical  techniques  such  as  multivariate, 
bivariate,  univariate  analyses  as  well  as  log-linear  analysis.  The  next  step  is  to 
normalize  the  data  in  order  to  compare  similar  systems. 

As  stated  in  Nussbaum  (2010),  data  normalization  reduces  content 

differences  such  as  “elements  of  cost;  technology  similarities,  and  consistency  in 

scope  of  work”.  However,  the  availability  of  historical  HSI  data  limits  data 

64 


normalization  for  cost  estimation.  When  comparing  similar  systems,  it  is  also 
important  to  break  down  activities  into  common  cost  elements.  For  example,  an 
estimate  in  one  program  may  contain  a  combination  SE/PM  cost  element  but 
historical  documents  list  these  elements  separately.  Division  of  cost  elements 
reduces  replication.  Mapping  the  differences  in  the  same  manner  in  estimation 
mitigates  the  likelihood  of  over-  or  under-estimating.  This  will  be  apparent  in  cost 
estimates  involving  HSI  criteria  since  many  process  owners  (HSI  domain  criteria) 
are  cost  oriented  this  way.  Since  HSI  elements  are  typically  under  current 
processes  (i.e.,  manpower,  training,  etc.),  this  will  inevitably  lessen  the  task  of 
creating  new  factors  to  explain  costs.  The  best  estimation  methods  rely  on  the 
analyst’s  judgment  and  the  data  available. 

The  current  thesis  investigates  methods  for  performing  estimates  involving 
HSI  criteria.  The  thesis  is  intended  as  a  resource  to  guide  the  HSI  modification  of 
current  cost  models  and  estimation  methods. 

C.  HSI  CONCEPT  MODEL 

The  purpose  of  this  section  is  to  describe  two  HSI  models  that  will  inform 
the  HSI  cost  estimation  process.  The  models  include  domain  variables, 
constraints,  enablers,  and  cost  drivers.  The  two  HSI  models  are  the  711*^  Human 
Performance  Wing’s  HSI  model,  (23  July  2007)  (see  Figure  16)  and  the  HSI 
Process  Model  developed  by  Miller  and  Shattuck  (2007)  (see  Figure  17). 
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Figure  16.  Linkages  between  the  HSI  process  model  and  the  Joint  Capabilities 
Integration  Development  System  gap  analysis.  (From  “Human  Systems 
Integration  (HSI)  in  Education,”  2007) 


The  Human  Performance  Wing  HSI  Model  (see  Figure  16)  is  described  as 
follows: 

The  four  input  domains  [of  HFE,  Personnel,  Training,  and 
Manpower]  greatly  simplifies  the  challenges  of  forecasting  the 
impact  of  HSI  trade-offs  through  modeling  and  simulation,  a 
necessary  consideration  given  DOD  initiatives  for  simulation-based 
acquisitions.  In  contrast,  the  ESOH,  habitability,  and  survivability 
domains  represent  desired  system  attributes  or  behaviors  not 
directly  procurable;  rather,  they  emerge  through  various 
combinations  of  the  input  domains.  (Tvaryanas,  Brown,  &  Miller, 

2009) 

Although  the  article  provides  an  adequate  assessment  of  the  constraints 
and  enablers  of  a  system,  it  does  not  represent  the  cost  variables  associated 
with  performance  optimization.  However,  the  model  developed  by  Miller  & 

Shattuck  (2007),  could  prove  useful  for  estimating  the  cost  of  HSI  activities. 
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Weighting  factors  or  cost  inputs  (relevant  to  previous  systems  seen  in  the  initial 
block  of  domain  advocacy),  initiate  processes  depicted  in  the  model.  The 
factors/inputs  that  could  augment  those  already  listed  in  the  Domain  Advocacy 
block  include  Manpower  Estimates,  Training  Technology  Development  costs. 
Modeling  and  Simulation  (M&S),  and  Test  and  Evaluation  (T&E)  costs.  The 
constraints  and  enabling  factors  of  the  model  include  schedule  delays,  contractor 
renegotiations,  performance  testing,  and  development  failures.  These  factors 
interact  with  the  inputs  to  optimize  the  system  outcome. 


System  Acquisition  Lifecycle  Environment  Including 
System  Requirements  and  Capabilities 


As  constrained  or 
enabled  by 
Development  Cost 
Domain  Delivery  Schedule 

Advocacy  for  Design  Risk 


M  +  P  +  T+HFE  +  Sur  + 
Hab  +  Sys  Saf  +  H  Haz 


(This  advocacy  can  take  (The  process  is 

various  forms,  e.g.,  influenced  by 

standards,  formulas,  these  contextual 

models,  regulations,  variables.) 

expertise.) 


Yields/Assessed/ 

Evaluated  by 

Usability 
Safety  Statistics 

Mishap  Rates  Total 

Survivability  ^  System 

Habitability  Performance 

Maintainability 
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Operating  and  Lifecycle 
Costs 


(Inputs) 


(Enablers  and  (Intermediary  or 
Constraints)  First  Order 

Outcomes) 


(Final 

Outcome/ 

Product) 


Figure  17.  HSI  Process  Model  (From  Miller  &  Shattuck,  2007) 


Miller  and  Shattuck  (2007),  state  as  “factors  in  the  first  two  boxes 
combine,  they  produce  effects  that  can  be  captured  by  outcome  measures  such 
as  usability,  survivability,  habitability,  maintainability,  reliability,  safety  (indicated 
by  safety  statistics  and  mishap  rates),  operating  and  total  life  cycle  costs”  (p.  12). 
The  outcome  measures  are  further  identified  by  cost  high  drivers  such  as 
deferred  compensation,  disability  cost  and  cost  avoidance  and  system 
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engineering  factors  (i.e.,  mean  time  between  failure,  maintenance  hours  etc.). 
The  product  of  all  equations  will  be  the  total  system  performance  of  the  human- 
system  product.  The  Miller  and  Shattuck  model  is  a  more  functional  model  (in 
terms  of  cost  estimations)  than  the  711  th  model. 

Most  expert  cost  estimators  agree,  “simpler  is  better,”  and  that  the  use  of 
sophisticated  tools  and  methods  of  cost  estimation  comes  with  its  own  burdens. 
Many  advanced  methods  utilize  “black  box”  applications  in  which  the  user  has 
little  knowledge  of  the  underlying  algorithms  or  processes.  Novice  estimators 
must  often  use  of  these  complicated  cost  estimation  tools  (e.g.,  ACEIT,  SEER-H, 
SEER-S,  PRICE-H,  PRICE-S,  etc.)  until  experience  is  gained.  This  thesis 
employs  a  relatively  simple  Microsoft  ®  Access  based  database  to  identify  HSI 
activities  and  their  associated  cost.  Excel-based  operations  perform  the 
necessary  activity  calculations.  This  thesis  will  use  the  HSI  process  model  by 
Miller  and  Shattuck  (2007).  However,  the  thesis  will  leverage  the  processes 
embedded  in  cost  estimation  tools  such  as  the  Automated  Cost  Estimating 
Integration  Tool  (ACEIT)  and  the  Marine  Corp  Ground  Training  Cost  estimation 
tool. 

Section  C  reviewed  HSI  Concept  models  for  future  cost  estimation 
processes.  Section  D  will  review  current  HSI  cost  estimation  methods  used  by 
other  countries  and  organizations. 

D.  CURRENT  SERVICE  HSI  COST  ESTIMATION  METHODS 

Reports  by  Canadian  Defense  Technology  Center  (DTC)  (2006);  Brooks, 
Greenley,  Dyck,  Salwaycott,  Scipione  and  Shaw  (2008);  and  Liu  (2009)  provide 
guidance  for  determining  a  method  to  perform  HSI  cost  estimation.  The 
Canadian  Ministry  of  Defense  initiated  the  DTC  HFI  program  in  2003,  which  is 
equivalent  to  U.S.  HSI. 

DTC  is  a  formal  collaborative  arrangement  between  industry  and 
academic  experts  in  a  particular  technology,  funded  jointly  by 
participants  and  the  MOD.  The  participants  work  together  to 
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generate  and  enhance  the  technology  vital  to  the  delivery  of  future 
UK  Defence  capabilities.  (British  Aerospace  Systems,  2010) 

The  Canadian  Defense  Technology  Center  Report  (2006)  identifies  five 
arguments  including  the  cost  considerations  of  five  categories  related  to  Human 
Factors  Integration  (HFI)  in  the  overall  cost  estimate  of  a  system  (see  Table  13). 
The  purpose  of  this  study  was  to  make  decisions  on  budget  allocation  regarding 
HFI  activities.  Due  to  a  misunderstanding  of  the  value  of  HFI  and  the  inability  to 
trace  intangible  benefits,  previous  work  had  not  provided  compelling  evidence  for 
including  HFI  into  cost  estimations.  The  arguments  lay  a  foundation  of  cost 
factors  and  provide  the  rationale  for  incorporating  HFI;  however,  the  bases  of 
these  cost  factors  are  sometimes  vague  or  obscure. 
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Table  13.  Human  Factor  Integration  Arguments 


Argument  1 

HFI  can  reduce  major  costs. 

Argument  2 

HFI  is  increasingly  required  due  to  developments  in 
technology. 

Argument  3 

HFI  plays  a  critical  role  in  identifying  and  mitigating 
operational  risks. 

Argument  4 

HFI  can  draw  on  resources  that  enable  an  effective  and 
efficient  process. 

Argument  5 

Effective  HFI  requires  early,  complete  and  close  involvement. 

The  report  identifies  the  value  of  HFI  contributions  by  providing  successful 
(and  non-successful)  case  studies  related  to  HFI  initiatives  (or  the  absence  of 
initiatives).  The  analyses  performed  on  these  case  studies  show  the  savings  or 
expenditures  related  to  specific  HFI  variables.  The  five  Arguments  and  their 
related  case  studies  are  now  described. 

Argument  1  includes  operational  and  developmental  risk  cost.  Main  costs 
include: 

•  Accident  costs  (system  safety  or  human  error), 

•  Lowered  performance  and  mission  failure,  people  and  materiel 
resources, 

•  Inefficient  development  and  implementation  processes  (Technology 
and  Development  phase;  Integration  phase  of  life  cycle), 

•  Redesign  cost  (integrate  phase),  and 

•  Personnel  costs  (Procurement-training  and  recruitment) 

Typical  case  studies  within  this  category  include  aviation  mishaps, 
equipment  designs  (rifles),  unanticipated  cost  due  to  extra  training,  improper 
implementation  and  late  design  changes.  In  some  cases,  it  is  hard  to  determine 
the  actual  cost  related  to  HFI.  For  example,  the  author  incorporates  full  cost  of 
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machinery  such  as  plane  loss,  but  omits  identification  of  hazards.  This  problem 
occurs  when  reporting  cause  of  failure.  It  is  important  to  perform  accurate 
tradeoff  analysis  and  identification  of  hazards  for  costs/health  hazards 
relationships. 

Argument  2  relates  to  complexity  driven  errors.  Increased  complexities  in 
technology,  mandated  standards  and/or  regulatory  actions  support  the  argument 
for  HFI.  Tradeoff  integration  should  provide  insight  into  complexity  related  errors; 
however,  the  costs  often  do  not  consider  these  tradeoffs. 

Argument  3  determines  the  need  for  identifying  and  mitigating  operational 
risk.  This  occurs  through  raising  awareness,  identifying  areas  of  uncertainty  and 
providing  a  means  for  user  input.  User-guided  input  leads  to  a  better  design  with 
less  rework.  A  valid  methodology  (through  modeling  of  potential  problems)  is  the 
key  to  cost  savings.  Early  identification  of  standards  and  substantiated 
requirements  that  can  be  used  later  in  scenario  development  also  saves  cost. 

Argument  4  indicates  that  HFI  practitioners  can  utilize  previous 
methodologies,  principles  and  research  to  prevent  repeating  past  mistakes.  The 
report  establishes  a  need  for  documenting  these  processes  to  mitigate  future 
Human  Factors  problems. 

Finally,  Argument  5  states  the  need  for  cost  estimation  throughout  the 
entire  process  of  the  life  cycle.  Four  methodologies  for  cost  estimation  include 
Investigating  (problems  and  constraints).  Creating  (solutions  based  on 
requirements).  Evaluating  (AoA),  and  Managing  (resources,  organization,  etc.). 
Furthermore,  the  argument  states  the  need  for  immersing  HFI  activities  early  in 
the  process.  This  will  allow  practitioners  to  develop  and  maintain  an  efficient  and 
cost  effective  solution. 

The  Defense  Technology  Centre  report  provides  excellent  examples  of 
HFI  problems  and  their  associated  costs.  The  report  also  provides  provisional 
insight  into  requirements,  validity  and  reliability,  increased  efficiency  and  the 
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need  for  user  involvement.  However,  future  work  should  include  detailed  analysis 
of  cost  factors  and  their  linkages  to  HFI  domains  across  the  life  cycle. 

Brooks  et  al.  (2008)  provides  a  second  HSI  cost  process.  The  Canadian 
Department  of  National  Defence  (DND),  in  partnership  with  Defense  Research 
and  Development  of  Canada,  provided  a  multi-year  analysis  of  activities  to 
demonstrate  and  validate  an  HSI  approach  for  the  Canadian  Materiel  Acquisition 
and  Support  community  of  DND. 

From  2001  to  2004  an  analysis  of  31  HSI-related  acquisition  projects  were 
completed  and  then  reviewed  for  cost  measures.  The  study  produced  measures 
in  terms  of  immediate  and  extrapolated  savings  based  on  the  HSI  activities  in 
these  projects.  The  study  also  provided  examples  of  future  budget  considerations 
such  as  incorporating  cost  estimation  ranges.  The  purpose  of  the  report  was  to 
establish  the  need  for  HSI.  The  report  included  the  following  topics:  concept  of 
HSI,  development  of  HSI  programs,  team  implementation,  integration  of  HSI 
process,  tools  of  HSI,  case  studies  (evaluation,  implementation,  support, 
definition  of  tools  and  lessons  learned)  and  cost  tracking  data  for  the  application 
of  HSI.  The  most  relevant  aspects  of  the  report  with  respect  to  this  thesis  are  the 
tool  applications  and  the  cost  tracking  data.  The  subsequent  paragraphs 
summarize  these  aspects. 

It  is  important  to  note  that  in  the  Canadian  DND,  HSI  consists  of  five 
domains.  In  the  U.S.,  the  number  of  domains  varies  from  one  military  service  to 
another  (U.S.  Air  Force,  nine;  U.S.  Navy,  eight;  U.S.  Army,  seven  domains).  The 
Canadian  DND  combines  the  manpower  and  personnel  domains  and  the  system 
safety  (SS)  and  health  hazards  (HHA)  domains.  The  SS-HHA  combination  is 
equivalent  to  the  U.S.  domains  of  system  safety,  occupational  health, 
survivability,  and  environment.  The  integration  of  the  domains  allowed  for  more 
centralized  control  over  projects  and  better  integration  of  HSI  activities  to  scale. 
Comparable  to  the  U.S.  approach  to  HSI,  the  DND  also  uses  MIL  HBK  46855A 
(HFE),  MIL  STD  882D  (SS).  In  addition,  the  DND  uses  CFITES  (the  standard  for 
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training  development).  In  order  to  perform  cost  estimations  for  U.S.  acquisition 
programs  using  Canadian  data,  these  composite  domains  and  their  cost  data  will 
have  to  be  decomposed. 

The  study  identified  three  categories  of  savings  in  the  31  case  studies. 
These  categories  can  be  applied  to  nearly  all  HSI  cost  estimation  activities. 
These  categories  included  immediate  savings,  extrapolated  savings  and 
uncalculated  savings.  These  three  measures  provide  a  useful  taxonomy  that  can 
be  employed  across  the  life  cycle  of  any  program.  The  article  described 
immediate  savings  as  items  or  resources  that  saved  time  and  money  during  the 
work  process.  The  extrapolated  savings  included  those  applications  of  HSI 
processes  that  occurred  early  in  the  life  cycle,  which  led  to  decisions  that  saved 
money  over  time.  The  third,  uncalculated  savings,  results  from  the  impact  of 
applying  HSI  rather  than  saving  lives  or  improving  operational  effectiveness. 

The  methodical  implementation  of  HSI  activities  within  the  Canadian  DND 
involves  several  documents  and  activities,  including:  an  HSI  plan.  Concept  of 
Operations  and  Concept  Support  (CONSUP);  Target  Audience  Descriptions 
(TAD);  System  Design  Inputs;  and  System  Evaluations.  Human  Factors 
Engineering  (HFE)  domain  activities  begin  with  a  task  analysis,  and  then  mission, 
function  and  task  decompositions  (backbone  of  system  safety  assessments). 
Next,  workload  analysis  and  predictive  studies  are  use  to  support  the  personnel 
domain  requirements.  Third,  the  results  of  the  detailed  task  analysis  are  used  to 
support  training  needs  assessments,  HHA  assessments  and  SS  assessments. 
Forth,  the  process  focuses  on  trade-off  analysis. 

The  results  of  the  analysis  in  the  report  indicated  that  these  projects  used 
that  HSI  practices  included  200  of  the  400  modeling  and  simulation  tools  that 
were  available.  These  tools  included  Human  Form  Computer  Aided  Drafting  and 
Design  (CADD)  tools,  prototyping,  simulation  tools,  task  network  modeling  (for 
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workload  analysis),  and  mission  tools,  as  well  as  function  and  task  analysis 
(MFTA)  tools.  Case  studies  focused  on  five  areas  of  analysis.  The  Brooks  et  al. 
(2008)  areas  included: 

•  HSI  program  development:  HSI  program.  Directorate  of  Technical 
Airworthiness  support  (DTA),  Modeling  and  Simulation 
Coordination  office  definition,  HSI  Concept  of  Operations,  TTCP 
HSI  workshop 

•  Major  HSI  processes:  Joint  Intelligence  information  Capability 
(JIIFC),  Advanced  Land  Fire  Control  System  (ALFCS),  Future 
Armored  Vehicle  System  (FAVS),  Multi  Mission  Effects  Vehicle 
(MMEV),  Maritime  Helicopter  Project,  MHP  modeling  and  workload. 
Very  Short  Range  Air  Defense  (VSHORAD) 

•  HSI  Process  subset:  Visual  Acuity  for  divers.  Grasshopper 
Uninhabited  Aerial  Vehicle  (UAV);  Canadian  Patrol  Frigate  (CPF) 
Accommodation,  Advanced  Linked  Extended  Reconnaissance  and 
Targeting  (ALERT)  Experimental  Design  Support) 

•  HSI  tool  evaluation:  Helmet  Mounted  Display  (HMD)  for  CF18, 
Clothe  the  Mounted  Soldier  (CMS)  survey.  Medium  Logistics 
Vehicle  Wheeled  (MLVW)  survey.  Surveillance,  Target  Acquisition, 
Night  Observation  (STANO)  survey  and  Collaborative  Displays. 

Provisions  of  HSI  and  project  definition  support:  CB  plus  Program  and 
performance  projection  framework,  Cipro  Plus  requirements.  Collaborative 
Planning  and  Management  Environment  (CPME),  Directorate  of  Training  and 
Education  Programs,  3D  modeling.  Nuclear  Biological  Chemical  Defense 
(NBCD)  Repository  Protection  Program,  Project  Activity  Reporting  System 
(PARS)  (Brooks,  2008). 

Through  pre-determined  criteria,  a  comparison  of  case  studies  produced  a 
likely  range  of  budget  percentages  (4-20%  HSI  cost  of  the  project  engineering 
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budget).  An  example  of  the  budgetary  considerations  included  the  designing  of 
“next  generation”  interfaces  for  fire  control  systems  and  experimental  programs. 
These  activities  provided  impact  assessment  for  crew  sizes  and  organizational 
structure.  Developmental  programs  applied  HSI  costs,  but  they  also  considered 
to  Commercial-off  the  Shelf  (COTS)  products.  Humans  Factor  and  Safety  will 
impact  COTS  and  system  integration  and  will  also  affect  the  installation  of  the 
system.  Program  considerations  were  not  the  only  concern.  The  subsequent 
paragraphs  describe  the  reports  discussion  of  the  role  of  cost  benefit  analysis. 

The  report  tracked  the  application  of  HSI  cost  benefits  to  determine  a  cost- 
benefit  framework  for  the  studies.  The  analysis  included  only  calculable  costs 
and  savings.  Of  the  five  categories  listed  previously,  only  categories  2  and  3 
were  considered  for  the  cost-benefit  calculations.  The  other  categories  did  not 
include  immediate  or  extrapolated  saving. 

An  example  of  a  cost-benefit  calculation  included  payback  amounts 
produced  by  HSI  applications.  By  removing  unnecessary  or  redundant 
functionality  early  in  the  design  process,  significant  savings  can  occur.  For 
example,  the  removal  of  redundant  displays  resulted  in  $5,000,000  worth  of 
savings  (Brooks  et  al.,  2010).  If  the  payback  or  savings  is  substantial,  the 
omission  is  a  success.  Uncalculated  savings  may  occur  when  design  changes 
positively  impact  the  quality  of  life  or  the  life  expectancy  of  the  Warfighter.  These 
savings  also  impact  safety  and  enable  effective  operations  (Greenley  & 
Associates,  2008).  However,  these  costs  were  not  used  in  a  total  cost-benefit 
analysis  because  of  their  multi-variant,  circular  and  discountable  characteristics. 
Future  analysis  should  include  these  costs.  The  report  suggests  that  if 
uncalculated  savings  can  be  estimated,  there  is  the  likelihood  the  cost  savings 
would  be  in  the  range  of  millions  of  dollars. 

The  report  follows  the  case  study  analysis  with  the  methodology  required 
to  demonstrate  the  cost  of  HSI.  The  data  produced  budgetary  percentages  for 
HSI  allocations.  The  percentages  of  HSI  investment  were  consistent  across 


75 


projects.  This  result  was  regardless  of  the  size  of  the  program.  The  changes 
were  a  function  of  the  HSI  activities  required  by  the  program  (i.e.,  human 
interface  project  complexity  initiated  increases  in  engineering  budgets). 

In  conclusion,  the  DND  report  provides  an  alternative  cost  estimation 
methodology.  It  attempts  to  demonstrate  a  method  for  integrating  the  various 
domains  of  HSI.  This  is  important  because  no  process  is  a  stand-alone  process. 
Trade-off  analysis  must  occur  across  the  domains  to  correctly  determine  all 
estimation  variables.  While  the  report  provides  excellent  information  on 
immediate  and  extrapolated  savings,  it  also  mentions  that  uncalculated  savings 
need  to  be  analyzed  more  specifically  which  will  result  in  exponential  savings. 

Liu  (2009)  explains  the  need  for  HSI  cost  measurement  by  examining 
Total  Ownership  Cost  reduction  in  Development  and  Procurement  stages  of  the 
life  cycle.  The  programs  of  choice  for  this  analysis  were  the  Unmanned  Aerial 
Systems  (UAS).  HSI,  System  Engineering  and  Program  Manager  costs  were 
minimal  compared  to  the  one  to  five  billion  dollar  RDT&E,  O&M,  Military 
Personnel  (MILPERS)  and  Procurement  budgets  for  the  UAS  programs 
examined.  Liu  described  the  current  estimation  method  as  a  “Rule  of  Thumb” 
measure.  This  approach  is  based  on  the  number  of  man-hours  and  the  total 
engineering  development.  In  the  “Rule  of  Thumb”  procedure,  historical  systems 
data  influence  the  estimate.  Factors  that  influence  the  cost  estimate  include 
expert  opinion,  technology  changes,  aircraft  weights  and  number  of  units.  The 
approach  used  in  the  article  was  parametric  estimation  methodology.  This  type  of 
methodology  incorporates  size  drivers  such  as  number  of  requirements, 
interfaces,  scenarios,  algorithms  and  volatility  factors.  The  method  also  uses 
effort  multipliers  (application  factors,  team  factors,  and  schedule  drivers). 
Together,  the  insertion  of  size  drivers  and  effort  multipliers  into  the  Constructive 
Systems  Engineering  Cost  Model  (COSYSMO)  modeling  tool  result  in  the  total 
System  Engineering  effort. 
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To  implement  the  model,  the  requirements  are  decomposed.  The 
requirements  decomposition  should  address  the  following: 

•  Determining  the  system  of  interest, 

•  Determining  whether  the  requirements  can  be  tested,  verified  or 
designed, 

•  Describing  the  relationship  of  the  system  of  interest  to  the  total 
system,  and 

•  Assessing  the  number  and  the  complexity  of  the  requirements 

Along  with  the  requirements,  the  effort  multipliers  are  established.  There 
are  14  effort  multipliers.  These  multipliers  include  issues  such  as  understanding 
requirements  and  architecture,  migration  complexity,  personnel,  tool  support,  and 
many  others.  High-level  integrated  process  teams  provide  these  multipliers. 

The  article  demonstrates  the  use  of  the  model  on  HSI  domain 
requirements  that  were  pre-determined  in  government  furnished  documents.  The 
author  uses  the  COSYSMO  model  to  describe  national  budget  percentages  for 
HSI  activities.  The  author  also  provides  suggested  required  inputs 
(variables/requirements  specified  earlier  in  this  thesis),  identifies  useful  outputs 
(identification  of  cost  drivers,  risk  issues,  and  discrepancy  warnings  early  in  the 
program),  and  describes  the  model’s  application  to  pre-Milestone  A  related 
activities.  The  Liu  (2009)  article  provides  a  better  understanding  of  requirement 
impacts  and  an  alternative  cost  estimation  approach  that  appears  to  be  more 
viable  than  the  traditional  methodology. 

E.  SUMMARY 

In  summary,  sections  (A)  through  (D)  discussed  the  decision  processes, 
concept  models,  and  methodologies  that  are  essential  to  formulate  a  new  HSI 
cost  estimation  approach.  HSI  and  cost  estimation  practitioners  can  use  this 
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approach  for  HSI  activities  throughout  the  acquisition  life  cycle  and  AoA.  The 
following  paragraph  summarizes  these  sections. 

Section  (A)  discussed  the  decision  support  system  and  the  Life  Cycle 
Management  System  HSI  activities.  This  section  addressed  the  HSI  activities 
(and  associated  cost  drivers)  that  occur  across  the  acquisition  life  cycle.  Section 
(B)  investigated  the  basic  cost  estimation  methods  for  performing  estimates 
involving  HSI  criteria.  The  section  discussed  the  Bratt  et  al.  (1998)  and  Valerdi 
(2005)  cost  methodologies  that  emphasized  HSI  criteria  adaptations  of  current 
cost  methods  in  Occupational  Health  and  System  Engineering.  These  methods 
provided  a  resource  to  guide  the  HSI  modification  of  current  cost  models  and 
estimation  methods.  Section  (C)  described  the  Miller  and  Shattuck  (2007)  and 
the  U.S.  Air  Force  711*^^  Human  Performance  Wing  (2006)  HSI  Models  that  will 
inform  the  new  HSI  cost  estimation  process  developed  in  this  thesis.  Section  (D) 
discussed  reports  by  Canadian  Defense  Technology  Center  (2006),  Brooks, 
et.al,  (2008),  and  Liu  (2009).  These  reports  provide  useful  insights  for 
determining  a  new  method  that  will  improve  the  cost  estimation  of  HSI  activities 
throughout  the  acquisition  life  cycle. 

The  literature  review  emphasized  the  need  to  identify  calculated  and 
uncalculated  savings,  direct  and  indirect  costs,  benefits,  risks,  and  trade-offs  as 
well  as  cost  variables  that  are  of  particular  interest  to  HSI  cost  estimation.  These 
sections  combined,  provided  the  necessary  background  to  develop  the  HSI 
activity  database  and  cost  models  presented  in  the  next  section  of  this  thesis. 

An  ultimate  goal  of  HSI  cost  estimation  will  be  the  ability  to  perform 
parametric  cost  estimates  with  HSI  historical  costs  (that  are  not  currently 
available).  Parametric  cost  methods,  that  use  approved  legacy  system  WBS, 
would  then  be  treated  as  baselines  that  could  then  be  incorporated  into  the 
estimates  for  funded  requirements.  The  development  of  an  HSI  activity  database 
is  necessary  to  perform  this  type  of  estimate. 
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The  next  chapter  introduces  the  first  iteration  of  the  proposed  HSI  Cost 
estimation  process  through  development  and  verification  of  the  HSI  database 
and  cost  model.  The  chapter  discusses  the  steps  taken  in  developing  the  HSI 
activity  database,  followed  by  the  initial  SME  Review  verification  studies.  Next, 
the  development  of  the  HSI  Cost  model  is  discussed.  Finally,  the  subsequent 
chapters  discuss  the  recommendations  for  future  verification,  validation,  and 
sensitivity  analysis  studies. 
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III.  METHOD 


A.  METHOD  OVERVIEW 

This  thesis  uses  an  exploratory  research  method  to  identify  those  HSI 
activities  in  the  acquisition  life  cycle  that  should  be  considered  when  performing 
AoA  cost  estimations.  A  database  of  these  activities  provides  a  framework  for 
future  historical  cost  estimation  processes.  The  study  elicits  Subject-matter 
experts  (SMEs)  feedback  to  determine  the  accuracy  and  supportability  of  the  HSI 
activities  in  each  of  the  life  cycle  phases.  The  feedback  is  then  used  to  modify 
the  HSI  activity  database  for  future  validation  studies,  as  well  as  to  promote 
better  cost  estimation  processes.  Furthermore,  the  current  HSI  cost  estimation 
methods  are  compared  in  terms  of  the  variables  they  consider. 

1.  Participants 

The  Navy  HSI  Working  Group  and  the  Air  Force  HSI  Office,  in  partnership 
with  the  Air  Force  711**^  Human  Performance  Wing,  supplied  the  SMEs  for  this 
study.  The  two  pre-selected  groups  of  SMEs  were  assigned  a  number  (one 
through  four)  that  corresponded  to  a  life  cycle  phase.  The  assignments 
accounted  for  SME  core  competency  areas  when  available  (e.g..  Requirements). 
Because  the  SMEs  were  participants  to  the  research  and  were  not  subjects  of 
the  research,  the  Naval  Postgraduate  School’s  Institutional  Review  Board  (IRB) 
determined  this  study  was  not  human  subjects  research. 

The  activity  review  was  designed  to  elicit  many  different  viewpoints  (i.e., 
holistic,  domain  specific).  SMEs  were  identified  as  experts  in  HSI,  an  HSI 
domain,  and/or  in  cost  estimation.  The  study  included  25  U.S.  Air  Force  and  16 
U.S.  Navy  respondents.  The  respondents’  communities  of  practice  and 
competency  areas  for  the  review  were  as  follows  (see  Figures  18  and  19).  The 
Navy  respondents  consisted  of  one  military  officer,  12  civil  service  employees. 
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and  3  government  contractors;  the  Air  Force  respondents  consisted  of  one 
military  officer,  15  civil  service  employees,  and  nine  government  contractors. 
Three  Navy  respondents  provided  cost  estimation  expertise.  However,  Air  Force 
cost  estimation  experts  were  unavailable  during  the  review  period. 

U.S.  Navy  SME 
Communities  of  Practice 


I  I  ■  ■  B 

Ns.:*l  Cost  Oth«r  SpiKO  i«n<l  ContrJKtor 

Systems  Estimation  ttaval  V<'arfare 

Command  Systems 

Comniand 


Figure  18.  U.S.  Navy  Communities  of  Practice 

U.S.  Air  Force  SME 
Communities  of  Practice 

8 
7 
6 
5 
4 
} 

2 
I 
0 


Figure  19.  U.S.  Air  Force  Communities  of  Practice 
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B.  SME  REVIEW  MATERIALS 


1.  Database  Development 

A  significant  part  of  this  thesis  involved  creating  a  database  of  HSI 
activities.  The  activities  were  extracted  from  research  articles,  government 
policies,  and  subject-matter  experts.  Comparison  of  existing  cost  estimation 
methods  provided  an  additional  source  of  HSI  activities  and  cost  drivers.  After 
compilation  of  these  activities,  an  initial  database  was  developed  (Database  is 
available  in  NPS  Electronic  Storage). 

The  database  to  be  use  during  the  SME  Review  was  designed  as  a 
Microsoft  ©Access  (HSI  activity)  database  that  was  exported  using  a  Microsoft  © 
Excel  file  format  (see  Figure  20). 


HSACOBvlX);  Database  (Access  2007)  -  Microsoft  Access 
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Figure  20.  HSI  Activity  Access  Database  (excerpt) 
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2. 


SME  Review  Design 


The  SME  review  consisted  of  an  Excel  workbook  containing  three 
spreadsheets  (i.e.,  Column  Heading  and  Descriptions,  Phase  Activities,  and  Cost 
Drivers)  (see  Appendix  C).  The  column  description  spreadsheet  provided  a 
proposed  list  of  database  column  headings  that  were  considered  important  to  the 
cost  estimation  process  (see  Figure  21). 
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Figure  21 .  SME  Review  HSI  Database  Column  Descriptions  (excerpt) 


The  activity  spreadsheet  included  98  (major)  life-cycle  phase  activities 
(MSA-24;  TD-23;  E&MD-22;  P&D-17;  0&S-12)  and  their  corresponding  sub¬ 
activities  (see  Figure  22).  Additionally,  the  spreadsheet  included  potential  HSI 
domains  and  cost  drivers  that  affected  each  major  activity. 
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Figure  22.  SME  Review  HSI  Activity  Spreadsheet  (excerpt) 
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The  final  spreadsheet  contained  a  generic  cost  driver  list  (see  Figure  23). 


Cost  Drivers 

Cognitive  Workload 

Physical  Workload 

Complexity 

Ratforms) 

CiioalityofTask 

Fhocess  capability  (CMMI  level  or  equivalent  rating) 

Difficulty  of  Task 

Requirements  understanding  (Subjective  assessment  of  the  system  requirements) 

Documentation  to  match  life  cycle  needs  (Breadth  and  depth  of  required  documentation) 

Size 

Frequency  of  Task 

SLOC(Software  lines  of  code) 

Initial  Training  (Instructor  &  Key  Ffersonnel) 

Specialty  Ffersonnel 

Initial  Training  (New  Equipment  at  system  fielding) 

Stakeholder  team  cohesion  (Subjective  assessment  of  all  stakeholders) 

Level  of  service  requirements  (Subjective  difficulty  of  satisfying  the  key  performance 
F^rameters) 

Target  Audience 

Maintenance  Concept 

Technology  risk  (Maturity,  readiness,  and  obsolescence  of  technology) 

Maintenance  Requirement 

Time  Constraint  for  System  Development 

Tool  support  (Subjective  assessment  of  SE tools) 

Migration  complexity  (Influence  of  legacy  system  (if  applicable)) 

Testing  Type-detailed  component  testing 

Testing  Type-subsystem  integration 

Modeling  &  Simulation 

Testing  Type-system  simulation  and  stimulation 

Multlsite  coordination  (Location  of  stakeholders  and  coordination  barriers) 

Testing  Type-campaign  or  force-level  (system  of  systems). 

Occupational  Training 

Type 

Operational  Capability 

Unit/Sustainment  Field  Training 

Ffersonnel  experience/continuity  (Subjective  assessment  of  staff  consistency) 

Warranties 

FfersonneLteam  capability  (Subjective  assessment  of  the  team’s  intellectual  capability) 

Figure  23.  SME  Review  Cost  Driver  List 


Finally,  a  workbook  was  assembled  that  contained  the  Column  Heading 
and  Cost  Driver  spreadsheets,  as  well  as  one  of  the  following  phases:  Materiel 
Solution  Analysis,  Technology  and  Development,  Engineering  &  Manufacturing 
and  Development,  and  combined  Production  and  Deployment  and  Operation  & 
Support. 

3.  Historical  Data  Collection 

Historical  system  data  also  were  collected  from  the  Air  Force  Total 
Operational  Cost  (AFTOC)  website.  Defense  Cost  and  Resource  Center 
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(DCARC),  and  the  ASN  (RDA)  website.  These  data  were  not  included  in  the 
present  study  but  could  very  well  be  used  in  future  validation  studies.  The  data 
included  RDT&E  product  development,  support,  and  test  &  evaluation  costs,  as 
well  as  O&S  manpower,  personnel,  and  training  costs.  The  specific  platforms  for 
which  data  were  available  included  the  Predator  (MQ-1),  Global  Hawk  (RQ-4), 
and  the  Broad  Area  Maritime  Surveillance  Unmanned  Aircraft  System  (BAMS). 

C.  SME  REVIEW  PROCEDURES 

1.  Distribution 

The  activity  database  review  was  administered  via  e-mail  to  pre-selected 
SMEs.  Each  participant  received  an  introductory  e-mail  explaining  the  purpose 
and  description  of  the  review  materials.  Next,  an  e-mail  containing  the  review 
materials  and  an  instruction  sheet  was  sent  (see  Appendix  D).  Respondents 
were  asked  to  complete  the  review  on  or  before  a  pre-determined  due  date  (21 
days  later).  Additionally,  each  respondent  received  a  reminder  e-mail  midway 
through  the  review  period. 

2.  Procedures 

SMEs  were  required  to  review  a  selected  (assigned)  portion  (a  particular 
phase  or  phases)  of  the  database  and  provide  feedback.  The  review  instructions 
were  as  follows.  SMEs  were  required  to  review  each  spreadsheet  to  determine  if 
the  data  were  appropriate  for  AoA  cost  estimates.  Next,  SMEs  were  required  to 
identify  any  data  modifications  (additions,  deletions,  corrections)  that  were 
required.  Each  spreadsheet  provided  blank  rows  for  the  SMEs  to  suggest 
missing  activities.  Furthermore,  SMEs  were  asked  to  estimate  the  percentage  of 
HSI  man-hour  effort  required  to  complete  the  activity.  SMEs  were  instructed  that 
each  of  the  phases  was  assigned  a  maximum  value  of  100%  man-hour  effort  for 
a  generic  ACAT  I  program.  Finally,  SMEs  were  asked  to  provide  an  overall 
percentage  of  HSI  man-hour  effort  for  the  entire  phase. 
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Once  the  review  was  completed,  the  SMEs  were  instructed  to  save  the 
document  using  an  Excel  (97-2003)  format  (“samefilename_yourinitials.xls” 
format)  and  to  e-mail  the  results  to  the  review  administrator. 

3.  Coding  Process 

Table  14  illustrates  the  coding  process  implemented  to  classify  database 
errors.  Any  data  block  that  received  a  CR,  INC  or  INCR  code  by  the  review 
administrator  were  modified  in  the  next  database  increment. 


Table  14.  SME  Review  Codes 


Code 

Category 

Explanation 

AA 

Additional  Activity 

SME  added  a  recommended  activity. 

C 

Correct 

No  (or)  minimal  correction  needed  (less  than  20%). 
Corrections  include  spelling  errors,  word  changes  or  activity 
item  changes.  The  Percentage  basis  is  the  total  number  of 
words  or  activity  action  items. 

CR 

Correct  with 
Recommendation 

SME  (or  Review  analyst)  considers  the  activity  correct  and 
the  SME  provides  a  recommendation  for  activity  adjustment 
(e.g..  Split,  re-sequence,  roll-up  ‘include  as  second  level  to 
another  activity’). 

E 

%  Effort* 

SME  provided  man-hour  effort  percentage  for  HSI  activity. 

GC 

General 

Comments 

SME  provided  a  question  or  statement  that  did  not  allow 
analyst  to  apply  another  code. 

INC 

Incorrect 

Multiple  corrections  needed  (greater  than  20%).  This 
percentage  is  based  on  number  of  words  or  activity  items. 

INCR 

Incorrect  with 
Recommendation 

SME  (or  Review  analyst)  considers  the  activity  incorrect 
and  the  SME  provides  a  recommendation  for  activity 
adjustment  (e.g..  Split,  re-sequence,  roll-up  ‘include  as 
second  level  to  another  activity’). 

NC 

No  Comment 

SME  did  not  respond  or  the  SME  has  considered  the  data 
appropriate. 

NQ 

Not  qualified 

SME  considered  themselves  inexperienced  to  comment  on 
the  activity. 

R 

Recommendation 

SME  provide  a  recommendation  for  correction  of  the 
activity  components. 

*Percent  Effort  was  based  on  100  %  for  each  phase. 
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D.  DEVELOPMENT  OF  HSI  COST  ESTIMATION  MODEL 

The  HSI  Cost  Estimation  Model  development  is  based  on  all  of  the  HSI 
activities  that  could  contribute  to  a  new  cost  estimation  process.  The  model  also 
combines  previously  validated  models,  cost  variables,  and  cost  drivers.  The 
review  administrator  applied  an  (S)  code  to  similar  method  components  and  a 
(DS)  code  for  dissimilar  components.  The  (DS)  coded  components  were 
recommended  for  further  review. 

As  stated  during  the  review  of  the  Valerdi’s  COSYSMO  model  (2005),  a 
cost  estimation  model  should  be  a  function  of  System  Engineering  man-hour 
effort  (see  Chapter  II,  Figure  11).  The  cost  estimation  model  for  this  thesis  is 
based  on  the  Valerdi  model,  but  is  a  function  of  HSI  effort  and  contains 
variables/drivers  that  are  elicited  from  other  methods  and  case  studies  (e.g., 
Canadian  Defense  methods  and  other  HSI  domain  methods).  The  following 
section  provides  the  review  results  and  the  proposed  model. 
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IV.  SME  REVIEW  RESULTS 


The  SME  Review  method  provided  response  rates,  adjustment 
percentages,  driver  types  and  HSI-Effort  percentages.  The  following  sections 
describe  these  results. 

1.  Activity  Review  Results 

Table  15  reports  the  overall  response  rate  by  type  of  respondent  and  total 
contact  count  (the  number  of  SMEs  that  received  the  review). 


Table  15.  Total  Response  Rate 


Total  Contacts 

83 

Total  Resiwndents 

40 

Overall  %  Response 

48% 

lotal  %  Government  Response 
(Military  aid  Civil  Service) 

70% 

Total  %  extractor  Response 

30% 

Table  16  contains  the  response  rates  by  service  and  phase.  The  response 
rates  ranged  from  14%  to  63%.  The  most  responses  were  received  for  the  MSA 
phase  (63%),  while  the  fewest  responses  were  received  for  the  TD  phase  (14%). 
Navy  rate  of  return  percentages  were  higher  overall,  however,  had  a  smaller 
sample  size  (USN=15,  USAF=25).  The  P&D  and  O&S  phases  resulted  in  the 
same  rates  of  return  (USN=57%;  USAF=46%). 
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Table  16.  Descriptive  Response  Rates 


Phase 

MSA 

T4D 

E&MD 

P&D 

o&s 

Phase  # 

1 

2 

3 

4 

c 

SeaiceGnouo 

US  AF 

USN 

US  AF 

USN 

USAF 

USN 

USAF 

USN 

US  AF 

USN 

Total  Phase  Contacts 

14 

8 

14 

6 

15 

6 

13 

7 

13 

7 

Total  Go  vemm  ent  CGO  V) 

8 

7 

10 

C 

10 

C 

8 

6 

8 

6 

Total  Co  ntractorlCTR) 

6 

1 

4 

1 

5 

1 

5 

1 

5 

1 

Total  Respondents 

5 

5 

2 

1 

6 

1 

6 

4 

6 

4 

Total  GOV 

3 

4 

2 

1 

3 

1 

4 

3 

4 

3 

Total  CTR 

2 

1 

0 

0 

3 

0 

2 

1 

2 

1 

H  T otal  Response 

36S 

63% 

14% 

17% 

40% 

17% 

46% 

57% 

46% 

57% 

%  GOV  Response 

38% 

57% 

20% 

20% 

30% 

20% 

50% 

50% 

50% 

50% 

%  Contractor  Response 

33% 

100% 

0% 

0% 

60% 

0% 

40% 

100% 

40% 

100% 

SME  recommendations  supported  the  accuracy  of  the  database.  Overall, 
the  adjustments  recommended  ranged  from  11%  to  24%  of  the  total  number  of 
entries  for  each  phase.  Table  17  provides  a  breakdown  of  the  review  phases,  as 
well  as  the  percentages  of  adjustments  required.  A  data  block  is  considered  to  be 
a  single  entry  into  the  spreadsheet  or  database.  The  Total  Activity  Data  Blocks 
column  consists  of  the  number  of  activities  (i.e.,  the  numbers  in  parentheses  in 
the  1®*  column)  multiplied  by  six  (the  number  of  columns  in  the  database).  The 
Total  Coded  Responses  consists  of  the  Total  Activity  Responses  multiplied  by  six 
(i.e.,  the  number  of  columns  in  the  database). 


Table  17.  Database  Descriptive  Breakdown  and  Accuracy  Rates 


Acquisition  Phase 
(#  Activities) 

Total  Activity 
Responses 

Total  Activity  Data 
Blocks 

Total  Coded 
Responses 

Total  Adjustments 
Required 

Percentage 

Adjustment 

Required 

MSA  (24) 

154 

144 

924 

183 

20% 

T&D  (23) 

39 

138 

234 

57 

24% 

E&MD  (22) 

82 

132 

492 

53 

11% 

P&D  (17) 

60 

102 

360 

74 

21% 

0&S(12) 

27 

71 

162 

19 

12% 

92 


Table  18  provides  a  breakdown  of  the  Total  Coded  Responses  (from 
Table  17)  by  coding  category  and  phase.  Table  18  also  provides  additional  detail 
with  respect  to  the  Total  Adjustments  Required  column  (in  Table  17).  The 
numbers  in  this  column  (of  Table  18)  are  based  on  the  following  coding 
categories:  Correct  with  Recommendation  (CR);  Incorrect  (INC);  Incorrect  with 
Recommendation  (INCR);  or,  General  Comment  (GC).  The  Percentage 
Adjustment  Required  (PAR)  results  consisted  of  dividing  the  Total  Adjustments 
Required  datum  by  the  Total  Coded  Responses. 


Table  18.  Activity  Code  Breakdown  by  Phase 


NC 

E 

C 

CR 

GC 

NQ 

INC 

INCR 

MSA 

464 

103 

167 

90 

42 

5 

2 

49 

T&D 

82 

46 

49 

32 

2 

0 

0 

23 

E&MD 

299 

44 

96 

25 

9 

0 

0 

19 

P&D 

163 

34 

89 

36 

26 

1 

0 

11 

O&S 

69 

23 

51 

6 

12 

0 

0 

1 

NC=No  Comment 

E=  Effort  Given 

C=Correct 

CR=  Correct  w/Recommendation 

GC=General  Comment  or  Question 
NQ=Not  Qualified  to  comment 
INC=lncorrect 

INCR=lncorrect  w/Recommendation 

2.  Database  Column  Review  Results 


SMEs  were  asked  to  review  the  database  columns  to  determine  whether 
they  would  provide  the  appropriate  information  necessary  for  establishing  a 
historical  cost  database  for  HSI  activities.  Each  SME  reviewed  19  data  blocks. 
Tables  19  and  20  provide  the  response  rates  and  coding  categories  for  this 
review  (see  Appendix  C  for  a  full  listing  of  columns  and  descriptions).  Overall,  the 
database  columns  received  positive  reviews.  SMEs  considered  all  columns 
useful  for  establishing  a  historical  database  in  support  of  HSI  cost  estimation 
activities.  The  data  blocks  that  required  adjustments  (30%)  were  the  based  on 
SME  general  comments  (GC)  or  recommendations  (CR)  for  enhancements  to  the 
database. 
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Table  19.  Database  Column  Response  Rate 


Total  Participants 

13 

Total  Responses 

124 

Total  Data  Blocks 

19 

Total  Coded  Comment  Blocks 

244 

Total  Adjustment  RQD 

72 

Percentage  Adjustment  RQD 

30% 

Table  20.  Column  Coding  Categories 


C 

CR 

INCR 

INC 

NC 

NQ 

GC 

51 

43 

0 

0 

120 

1 

29 

3.  Cost  Driver  Section  Review 

The  following  list  provides  the  results  from  the  review  of  the  cost  drivers. 
SMEs  commented  on  the  accuracy  of  the  proposed  cost  drivers  and  classified 
them  by  cost  type  (i.e.,  (D)irect  or  (l)ndirect).  Overall,  the  list  of  cost  drivers 
appeared  to  be  adequate.  However,  some  SMEs  noted  that  certain  drivers  were 
specific  to  HSI  activities  while  other  drivers  could  be  applied  to  many  other 
acquisition  activities.  Respondents  suggested  that  three  drivers  be  removed 
(Size,  Platform  and  Type).  One  of  these  drivers  (type),  however,  was  acceptable 
upon  correction  to  the  term  system  type. 
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Cost  Driver  Results  List 


(I)  Cognitive  Workload 

(I)  Complexity 

(I)  Criticality  of  Task 

(I)  Difficulty  of  Task 

(I)  Frequency  of  Task 

(I)  Level  of  service  requirements 

(Subjective  difficulty  of  satisfying 

the  key  performance 

Parameters) 

(I)  Mission 

(I)  Modeling  &  Simulation 
(I)  Multisite  coordination 
(Location  of  stakeholders  and 
coordination  barriers) 

(I)  Personnel 

experience/continuity  (Subjective 
assessment  of  staff  consistency) 
(I)  Personnel/team  capability 
(Subjective  assessment  of  the 
team’s  intellectual  capability) 

(I)  Physical  Workload 
(I)  Process  capability  (CMMI 
level  or  equivalent  rating) 

(I)  Requirements  understanding 
(Subjective  assessment  of  the 
system  requirements) 

(I)  Stakeholder  team  cohesion 
(Subjective  assessment  of  all 
stakeholders) 

(I)  Target  Audience 
(I)  Technology  risk  (Maturity, 
readiness,  and  obsolescence  of 
technology) 

(I)  Time  Constraint  for  System 
Development(l)  Tool  support 
(Subjective  assessment  of  SE 
tools) 


(D)  Documentation  to  match  life 
cycle  needs  (Breadth  and  depth 
of  required  documentation) 

(D)  Initial  Training  (Instructor  & 
Key  Personnel) 

(D)  Initial  Training  (New 
Equipment  at  system  fielding) 
(D)  Maintenance  Concept 
(D)  Maintenance  Requirement 
(D)  Manning 
(D)  Migration  complexity 
(Influence  of  legacy  system  (if 
applicable)) 

(D)  Occupational  Training 
(D)  Operational  Capability 
(D)  SLOC 

(D)  Specialty  Personnel 
(D)  Testing  Type-detailed 
component  testing 
(D)  Testing  Type-subsystem 
integration 

(D)  Testing  Type-system 
simulation  and  stimulation 
(D)  Testing  Type-campaign  or 
force-level  (system  of  systems) 
(D)  Unit/Sustainment  Field 
Training 
(D)  Warranties 
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In  addition  to  the  classification  of  the  listed  drivers,  the  SMEs  provided 
additional  drivers  as  seen  in  the  following  list.  The  list  was  further  classified  by 
the  administrator  to  include  cost  type  (i.e.,  direct  (D)  or  indirect  (I))  based  on  the 
Federal  Accounting  Regulations  classification  process. 


(D)  Facilities  -  Development  and 
test 

(D)  Facilities:  Operations  and 

maintenance 

(D)  Facilities:  Training 

infrastructure 

(D)  Force  Structure  Composition 
(Active,  Reserve,  Officer, 
Enlisted,  Civilian,  Contractor) 

(D)  Safety  requirements  and 
regulations 

(D)  Special  tools  &  equipment: 
development; 

operations  and  maintenance 

(D)  Specialized  tools  required  for 

maintenance 

(D)  Training  location 

(I)  Habitability  shortfalls  that  lead 

to  workforce  attrition 

(I)  Accessibility 

(I)  Complexity  (sub-categories): 

technological  complexity  & 

organizational  complexity 

(I)  Configuration  and  Change 

Management 

(I)  Consequence  of  errors 

(I)  CCTS  and  GCTS  suitability 

assumptions 

(I)  Errors  identified 

(I)  Facility  layout 

(communication,  distance  to 

amenities,  distance  from  noise 

source) 

(I)  Function  allocation  (human, 
machine,  supervisory,  manual) 


(I)  Intuitive  usability  (can  novices 
operate  without  any  training) 

(I)  Material  selection 

(I)  Modeling  &  Simulation 

Validation 

(I)  Poor  Situation  Awareness 

(I)  Poor  usability 

(I)  Required  KSAs 

(I)  Requirements  Determination 

(I)  Situational  awareness  of 

automation  state 

(I)  Specialized  safety 

(I)  Specialty  Personnel 

(development) 

(I)  Team  design 
(I)  Test  safety 

(I)  Training  Analysis  and  Design 
(I)  Training  pipeline  (or  other 
training  "time"  metric) 

(I)  Under-manning 
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4. 


HSI  Effort  Results 


The  final  analysis  consisted  of  looking  at  percentages  of  HSI-Effort  across 
the  life  cycle  phases  for  a  general  ACAT  I  program.  The  results  in  the  following 
sections  are  intended  to  provide  a  starting  point  or  baseline  for  HSI  costs.  The 
results  in  Tables  21  through  25  are  ordered  according  to  the  acquisition  cycle 
phases.  A  paper  format  precludes  presentation  of  the  entire  database.  Therefore, 
the  tables  show  only  the  first  two  lines  of  each  row.  A  complete  database  is 
available  in  electronic  format  upon  request.  Additionally,  each  table  displays  the 
total  number  of  SME  participants  for  each  phase.  Furthermore,  the  data 
presented  in  these  tables  is  non-normalized. 

Table  21  provides  the  Materiel  Solution  Analysis  Phase  HSI-Effort  results. 
Pre-Materiel  Solution  Analysis  (or  Pre-Materiel  Development  Decision)  activities 
were  provided  an  event  coding  of  (A)  through  (F)  while  the  Materiel  Solution 
Analysis  activities  are  coded  (G)  through  (S).  The  results  ranged  an  average  of 
3%  to  28%  for  the  various  activities.  In  other  words,  SMEs  estimated  these 
activities  would  represent  between  3%  and  28%  of  the  total  (i.e.,  100%)  HSI 
effort  for  the  MSA  phase.  The  highest  HSI  activity  percentages  were  reported  for 
the  following  activities: 

•  Provide  HSI  activity  input  during  Capabilities  Based  Assessment  for 
ICD  (M=23%,  SD=0.20); 

•  Provide  an  HSI  assessment  based  on  the  best  available  solution 
descriptions  (in  CBA,  AoA,  Study  Guidance,  ICD)  across  all  HSI 
domains  and  to  define  the  risk  level  in  each.  (M=28%,  SD=0.21); 

•  Provide  a  total  system  analysis  based  on  functional  relationship 
perspectives  with  an  emphasis  on  system  boundaries  (M=13%, 
SD=0.11);  and, 

•  Participate  in  the  AoA  assessment  (M=10%,  SD=0.02). 
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Furthermore,  USN  reviewers  suggested  combining  the  domains  of  Environment, 
Occupational  Health  and  Safety  while  USAF  reviewers  replaced  Occupational 
Health  with  Health  Hazards. 


Table  21.  MSA  HSI-Effort  Percentage  (based  on  input  from  10  SMEs) 


Event 

HSI_Activity 

tiEAH% 

mit 

pucdon 

STDV 

A. 

Provide  HSi  activity  input  during  Capabilities  Based 
Assessment  forICD 

23% 

0.20 

B. 

Provide  HSI  input  into  the  HSIP  section  o''  the  System 
Engineering  Plan  (SEP) 

8% 

0.03 

c. 

Provide  a  total  system  analysis  based  on  functional 
relationship  perspectives  with  an  emphass  on  system 

13% 

0.11 

D 

Provide  HSI  considerations  into  tie  AoA  Study  plan 

3% 

0.02 

E. 

Provide  HSI  input  into  ESOH  Integration  Strategy. 

6% 

0.04 

F. 

Perform  HSI  assessments  based  on  best  availabe  solution 
description  (in  CBA,  AoA,  Study  Guidance.  ICD)  across  a  1 

28% 

0.21 

E. 

Perform  an  H  SI  analysis  of  legacy  systems  with  respect  to 
each  domaih  ard  addressed  to  potential  integrated  impact 

8% 

0.04 

F 

Contribute  to  assessment  and  development  of  system 
concepts  /Materiel  solutions. 

4% 

0.01 

G. 

Assist  in  development  of  cost  estimate  besed  on  anticipated 
HFE,  Manpower.  Personnel.  Traninq,  safety  activities/ 

4% 

0.01 

H. 

Conduct  Mission  Task  Analysis. 

7% 

0.04 

1 

Conduct  a  Initia  Top  Down  Reqcirements  Analysis. 

8% 

0.02 

J. 

Assist  in  performance  of  Manpov/er  /Personnel  alternatives 
and  concept  developmeit  and  assessments 

4% 

0.01 

K 

Identify  Manning  needs,  manpower  lessons  learned,  and  high 
driversfrom  legacy  systems. 

4% 

0.02 

L 

Assist  in  linear  modeling  and  studies  analysis  of  system 
approach  alternatives. 

7% 

0.02 

M. 

Conduct  Manpower/Trailing  assessments  to  determine 

KPPs  learning  cbjectives  and  training  reguirements  for 

6% 

0.04 

N. 

Assist  in  deriving  systematic  preciction  of  conditions  through 
Initial  Safety,  and  Occupational  Health  requirements  and 

4% 

0.01 

O. 

Participate  in  AoA  assessment 

10% 

0.02 

P, 

Define  concept  capabilit  es  and  requirements 

3% 

0.02 

Q. 

Provide  HSI  input  into  Technology  Development  Strategy 

3% 

0.00 

R. 

Provide  HSI  Input  into  Life  Cycle  Sustainment  Plan. 

4% 

0.02 

S. 

Provide  HSI  input  into  Support  System  Engineering 

Technical  Reviews  (SETRs) 

3% 

0.01 
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Tables  22  and  23  provide  the  HSI  Effort  for  the  Technology  Development 
phase.  For  this  phase,  SMEs  estimated  these  activities  would  represent  between 
3%  and  20%  of  the  total  (i.e.,  100%)  HSI  effort.  Several  of  the  activities  that  are 
listed  (callout/  guide  or  new  activities  suggested  by  SMEs)  have  only  one  SME 
effort  contribution.  These  values  have  a  standard  deviation  of  zero. 

The  highest  HSI  Effort  percentages  were  attributed  to  the  following  activities: 

•  Determine  the  need  for  an  HSI  IPT  or  working  group  (M=20%, 
SD=0.00), 

•  Demonstrate  &  validate  System  &  Technology  maturity  versus 
defined  user  needs  &  environmental  constraints  (M=20%,  SD=0.00) 

•  Decompose  functional  definitions  into  Critical  Component 
Definitions  &  Technologies  Verification  Plan  (M=16%,  SD=0.00) 

•  Demonstrate  enabling/critical  Technology  components  versus  plan 
(M=13%,  SD=0.08), 

•  Perform  analysis  and  recommendations  for  system  technology 
development  (M=12%,  SD=0.12), 

•  Provide  HSI  input  into  the  Capability  Development  Document  and 
Acquisition  Baseline  (M=11%,  SD=0.08;  M=11%,  SD=0.08),  and 

•  Conduct  formal  development  of  human-centered  source  selection 
criteria  for  RFPs  at  Milestone  (MS)  A  (M=11%,  SD=0.13). 
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Table  22.  Technology  Development  Phase  HSI  Effort 
(based  on  input  from  three  SMEs) 


EVENT 

HSI_Activity 

MEAN  Activity 
Effort 

SD 

GUIDE 

Review  all  the  inputs  to  the  Technology  Development  Phase 
including  the  following: 

20% 

(one 

value) 

A.  New 
Activity 

Determine  the  need  for  an  HSI  IPT  or  a  working  group 

20% 

(one 

value) 

B. 

Establish  a  HSI  IPT  or  Working  Group  -  consistent  with  the 
size  and  complexity  of  the  envisioned  acquisition  program 

10% 

0.03 

GUIDE 

Throughout  the  next  5  steps  C-G  conduct  trade  offs  and 
support  the  AoAs  with  the  identified  sub  activities. 

C. 

Determine  all  relevant  Critical  Operational  Issues  and 
determine  and  develop  Measures  of  Effectiveness 

5% 

0.01 

D. 

Callout 

Develop  System  Performance  (&Constraints)  Specification 
&  Enabling/Critical  Technologies  &  Prototypes  Verification 

10% 

(one 

value) 

E. 

Callout 

Develop  Functional  Definitions  for  Enabling/Critical 
Technologies/Prototypes  &  Associated  Verification  Plan 

10% 

(one 

value) 

F. 

Callout 

Decompose  functional  definitions  into  Critical 

Component  Definition  &  Technologies  Verification  Plan 

16% 

(one 

value) 

G. 

Identify  HSI  technical  risks  and  mitigation  strategies  for 
technology  development. 

10% 

(one 

value) 

GUIDE 

Throughout  the  next  7  Steps  H-0  conduct  trades  that  are 
spread  across  the  identified  sub  activities. 

H. 

Demonstrate  enabling/critical  technology  components 
versus  plan 

13% 

0.08 

I. 

Provide  integrated  HSI  assessment  methodology  for 
technology  down  selection  recommendations  and  use  in  full 

7% 

0.05 

J. 

Refine  and  update  requirements 

10% 

0.00 

K. 

Assist  in  technology  selection 

7% 

0.05 

L. 

Perform  analysis  and  recommendations  for  system 
technology  development. 

12% 

0.12 

M. 

Assist  in  development  of  System  Functional  Specs  & 
Verification  Plan  to  evolve  system  functional  baseline 

6% 

0.06 

N. 

Participation  in  Integrated  Test  Team  (ITT)  and  Evaluation 
or  other  Integrated  Performance  Team  IPT  as  may  be 

8% 

0.03 

100 


Table  23.  Technology  Development  Phase  HSI  Effort  (continued) 


EVENT 

HSI_Activity 

MEAN  Activity 
Effort 

SD 

Guide 

Throughout  the  next  7  Steps  P-S  conduct  trades  that  are 
spread  across  the  identified  sub  activities. 

20% 

(one 

value) 

P. 

Callout 

Demonstrate  enabling/critical  technology  components 
versus  plan 

16% 

(one 

value) 

Q. 

Callout 

Demonstrate  system  &  prototype  functionality  versus  plan 

10% 

(one 

value) 

R. 

Callout 

Demonstrate/model  integrated  system  versus  performance 
specification 

10% 

(one 

value) 

S. 

Callout 

Demonstrate  &  validate  system  &  technology  maturity 
versus 

20% 

(one 

value) 

GUIDE 

Insure  the  production  and  completion  of  inputs  to  the 
following  products: 

T. 

Review  and  provide  input  into  Preliminary  design  and 
allocation  baseline. 

8% 

0.10 

U. 

Provide  HSI  input  into  the  Capability  Development 

Document 

11% 

0.13 

V. 

Provide  input  into  the  Acquisition  Strategy  for  goals, 
objectives  and  entrance  and  exit  criteria  as  a  member  of 

3% 

0.03 

W. 

Provide  input  into  Acquisition  Program  Baseline  (APBs) 

11% 

0.13 

X. 

Incorporate  requirement  for  contractor  generated  HSIP(s) 
and  associated  source  selection  criteria  in  Request  for 

8% 

0.03 

Y. 

Conduct  formal  development  of  human-centered  source 
selection  criteria  for  RFPs  at  Milestone  (MS)  A 

11% 

0.13 

Z. 

Apply  analysis  in  findings  to  Test  and  Evaluation  Strategy 
(TES),  LCMP  and  TEMP 

6% 

0.06 

AA. 

Provide  input  into  System  Engineering  Review  process 
(SRR). 

7% 

0.04 

BB 

Provide  input  into  System  Engineering  Review  process 
(SFR) 

10% 

(one 

value) 

CC 

Provide  input  into  System  Engineering  Review  process 
(PDR) 

16% 

(one 

value) 
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Table  24  provides  the  HSI  Effort  for  the  Engineering  and  Manufacturing 
Development  Phase.  The  results  for  these  activities  ranged  from  2%  to  8%  of  the 
total  HSI  effort.  Activities  that  resulted  in  the  same  percentage  contributions  (i.e., 
5%, 5%,  5%)  display  a  standard  deviation  of  zero.  The  HSI  activities  with  highest 
effort  percentages  included: 

•  Assisting  in  development  of  test  plans  for  integrated  system  testing 
{M=8%,  SD=0.03);  and, 

•  Providing  HSI  input  into  technical  reviews  and  Systems 
Engineering  activities  {M=8%,  SD=0.04). 

SMEs  suggested  moving  two  activities  from  the  MSA  phase  to  the  TD  phase. 
These  activities  pertained  to  proving  HSI  input  into  the  draft  CDD  and  the 
Training  Program  Plan. 


Table  24.  E&MD  Phase  HSI  Effort  (based  on  input  from  seven  SMEs) 


Event 

Category 

HSI _ >Vcti  vity 

MEAN  %  Effort 

SD 

/K. 

PrcDN/ide  HSI  inpul:  to  the  “TEfS/IP 

5% 

0.00 

B. 

Finalize  HSI  Rec|  ui  i  re  me  n  ts  toy 
ototaining  agreement  from 

2% 

0.01 

c. 

Assist  in  development  of 
detailed  design  rec|  lj  i  re  me  n  ts 

2% 

0.02 

D. 

Assess  detailed  system 
designs  to  ensure  that  they 

4% 

0.02 

E. 

Participate  in  prototypes/test 
materials  development  (for 

5% 

0.04 

f=. 

Evaluate  prototypes  (or  test 
materials)  for  HSI  Criteria 

ISJ  e  w 
Activity 

G. 

Participate  in  Design 
Demonstrations  (to  include 

0.03 

H  . 

Evaluate  HSI  criteria  for  the 
system  during  component  / 

5% 

0.00 

Perform  HSI  evaluation  on 
System  Components 

4% 

0.02 

J  . 

Assist  in  Development  of  Test 
Plans  for  integrated  system 

0.03 

K. 

Participate  in  HSI  test  events 

0.02 

I_. 

Participate  in  ECP 
development 

3% 

0.02 

IVI . 

Provide  HSI  updates  to 
Acquisition  Strategy  and 

2% 

0.01 

ISI . 

Provide  HSI  input  into 

Ca  pa  toi  1  ity  P  rod  u  ct  io  n 

4% 

0.01 

O. 

Provide  HSI  input  into 
technical  reviews  and  Systems 

S% 

0.04 

R*. 

Provide  HSI  input  for  error  and 
fault  analysis 

3% 

0.01 
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Table  25  provides  the  HSI  Effort  for  the  Production  and  Development 
phase.  The  events  coded  (A)  through  (N)  ranged  from  an  average  of  4%  to  10% 
of  the  total  HSI  effort.  The  highest  HSI  effort  percentages  were  reported  for  the 
following  activities: 

•  Test  and  Evaluation  support  and  participation  (LFT&E)  (M=10%, 
SD=0.07); 

•  Assisting  with  the  System  Hazard  Analysis  (SHA)  and  Operating 
and  Support  Hazard  Analysis  {M=8%,  SD=0.04);  {M=^3%, 
SD=0.11);  and, 

•  Participate  in  the  AoA  Assessment  (M=10%,  SD=0.02). 

SMEs  suggested  the  insertion  of  three  additional  activities. 


Table  25.  Production  and  Deployment  Phase  HSI  Effort 
(based  on  input  from  10  SMEs) 


Event 

HSI_Activity 

Mesn  Activity 
Effort 

Msn  Hours) 

SD 

A. 

HSI  PIsnning  snd 

5% 

0.01 

B. 

Review  end  Develop  HFE 

T% 

0.04 

C. 

Verify  HSI  requirements  for 

4% 

0.01 

D. 

Assist  in  T&E  HSI  snelyses  of 
system  design  end  instelletion 

0.02 

E. 

Assist  in  T&E  HFE  enelyses  of 
system  design  end  instelletion 

e% 

0.01 

F. 

T&E  support  end  perticipetion 

e% 

0.03 

G. 

T&E  support  end  perticipetion 
(LFT&E) 

1  o% 

O.OT 

H. 

T&E  support  end  perticipetion 
(FOT&E) 

r% 

0.02 

1. 

Review  OPTEVFOR  reports 

5% 

0.00 

J. 

Document  end  enelyze  post 
deployment  lessons  leerned 

5% 

0.01 

K. 

Provide  input  into  Review 
process  for  P&D  Phese 

4% 

0.02 

L. 

Updete  Menpower/Personnel 
requirements  for  system 

e% 

0.01 

M. 

Updete  Treining  systems 
requirements  for  system 

e% 

0.00 

N. 

Assist  with  System  Hezerd 
Anelysis  (SHA)end  Opereting 

8% 

0.04 
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Table  26  provides  the  HSI-Effort  contributions  for  the  Operation  and 
Support  phase.  The  results  for  activities  (A)  through  (K)  ranged  from  5%  to  15% 
of  the  total  HSI  effort.  The  highest  HSI  activity  percentages  reported  for  the 
activities  included: 

•  Provide  Follow-on  Test  and  Evaluation  support  {M=^5%,  80=0.07)] 
and, 

•  Provide  testing  and  verification  of  Follow-on  Test  &  Evaluation 
processes  {M=^2%,  80=0.03). 

SMEs  suggested  incorporating  two  of  the  original  database  activities  into 
activities  that  pertained  to  additional  Manpower  or  HFE  support  during  Full 
Operational  Capability  Support.  SMEs  did  not  remove  any  HSI  activities  from  the 
database  but  did  add  one  activity  (HSI  planning  and  management). 


Table  26.  Operation  and  Support  Phase  HSI  Effort  (based  on  input  from  10  SMEs) 


Event 

HSI_Activity 

MEAN  %  Effort 
(Man  hours) 

ST  DEV 
(%  EFFORT) 

A. 

HSI  Flanning  and 

IVIanagennent 

8% 

New 

Activity 

B. 

Provide  Follow-on  Test  and 
Evaluation  support 

1  5% 

0.07 

C. 

Provide  analysis  for  HFE  FOC 
support  and 

8% 

0.01 

D. 

Provide  recommendations  for 
HFE  FOC  support  and 

8% 

0.03 

E. 

Provide  Training  FOC  support 
and  logistics/sustainment 

9% 

0.01 

F. 

Provide  IVIanpower/Personnel 
FOC  support  and 

6% 

0.01 

G. 

Analyze  and  prototype  post¬ 
product  improvements  for  the 

7% 

0.02 

H. 

Provide  input  for  sustainment 
of  operational  readiness  and 

7% 

0.04 

i- 

Provide  testing  and  verification 
of  Follow-on  T&E  processes. 

1  2% 

0.03 

J. 

Provide  FOC  support  and 
logistics/sustainment 

8% 

0.03 

K. 

Provide  input  into  O&S  review 
processes  (ISR) 

5% 

0.01 
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5.  Results  Summary 

The  SME  Review  provided  results  that  included  the  Descriptive  and  Total 
Response  Rates,  Database  Adjustment  Rates,  Cost  Driver  use  and  type 
determination  and  HSI-Effort. 

This  thesis  returned  an  overall  response  rate  by  service  and  type  of 
respondent.  The  most  of  the  responses  occurred  during  the  MSA  phase  (63%), 
while  the  fewest  responses  were  received  for  the  TD  phase  (14%).  Overall,  the 
adjustments  recommended  ranged  from  11%  to  24%.  The  most  adjustments 
occurred  during  the  TD  phase  (24%),  while  the  E&MD  phase  received  the  fewest 
adjustments  (11%). 

The  second  results  section  pertained  to  the  HSI  activity  database  columns 
that  are  required  to  establish  a  HSI/cost  estimation  database.  The  SMEs 
considered  all  columns  were  useful  for  establishing  a  historical  database  in 
support  of  HSI  cost  estimation  activities. 

The  third  section  of  results  pertained  to  review  of  cost  drivers.  These 
results  provided  an  identification  of  drivers  (type)  as  well  as  33  additional  drivers 
that  can  attribute  to  an  HSI  cost  estimation  process. 

Finally,  the  SMEs  provided  HSI-Effort  percentages  of  the  life  cycle  phases 
for  a  general  ACAT  I  program.  These  results  are  intended  to  provide  a  starting 
point  or  baseline  for  HSI  costs.  The  HSI-Effort  (out  of  100%)  results  included  the 
following  percentage  ranges:  MSA  (3-28%),  TD  (3-20%),  E&MD  (2-8%),  P&D  (4- 
10%),  and  O&S  (5-15%).  The  remainder  of  this  thesis  discusses  these  results  in 
more  detail  as  well  as  provide  recommendations  for  future  work. 
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V.  DISCUSSION 


The  purpose  of  this  thesis  was  to  develop  a  qualitative  method  for 
determining  HSI-specific  costs  that  can  be  used  for  conducting  tradeoffs  during 
the  AoA.  The  effort  was  necessary  to  improve  the  accuracy  of  cost  estimates 
over  the  life  cycle  of  the  system  and  to  provide  a  basis  for  a  new  HSI  cost 
estimation  process.  The  method  produced  an  HSI  activity  database,  HSI  cost 
drivers  and  an  HSI  level  of  effort  for  every  acquisition  life  cycle  phase  for  a 
generic  ACAT  I  program  (See  Appendix  C).  SMEs  provided  the  requisite 
knowledge  for  refining  the  database.  The  following  sections  discuss  the 
qualitative  findings. 

A.  RESPONDENTS 

Although  the  sample  size  was  small,  the  range  of  SME  competency  areas 
provided  a  diverse  view  of  the  HSI  cost  processes.  HSI  is  a  relatively  young 
discipline  with  few  SMEs.  The  small  pool  of  SMEs  (who  tend  to  be  in  high 
demand  and  are  typically  overworked)  may  have  contributed  to  the  somewhat 
low  response  rate  of  approximately  48%.  Other  factors  that  contributed  to  the 
response  rate  included  availability,  unfamiliarity  with  phase  assignment,  and 
confusion  concerning  what  was  expected  of  the  SME. 

Response  rates  varied  within  and  between  services  for  several  reasons. 
For  example,  USAF  SMEs  were  reassigned  during  the  review  and  were  unable 
to  complete  their  assessment  of  the  database.  In  these  cases,  respondents  were 
encouraged  to  provide  as  many  recommendations  as  time  permitted.  Therefore, 
analyses  were  based  on  individual  data  item  entries  instead  of  individual 
responses. 

Every  attempt  was  made  to  match  a  SME’s  expertise  to  the  appropriate 
acquisition  phase.  However,  some  SMEs  reported  they  were  unfamiliar  with  the 
HSI  activities  within  the  acquisition  phases  to  which  they  were  assigned.  In  these 
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cases,  when  SMEs  appealed  to  the  review  administrator,  they  were  offered  an 
alternate  phase  to  review.  Respondents  also  gave  domain  viewpoints  (i.e., 
training,  manpower)  as  well  as  holistic  viewpoints. 

B.  HSI  ACTIVITY  DEVELOPMENT 

The  HSI  database  responses  provided  validation  of  the  activities  needed 
to  perform  cost  estimations.  During  the  HSI  activity  evaluation,  SMEs 
emphasized  balancing  the  results  of  this  study  of  the  activities  found  in  the 
system  engineering  process  and  human  view  frameworks.  Additionally,  SMEs 
suggested  that  future  efforts  should  clarify  HSI  activities  in  the  Acquisition 
Management,  Acquisition  Engineering  and  Support/Sustainability  activity  tiers. 
This  clarification  would  necessitate  determining  “areas  of  responsibility”  within 
the  HSI  domain  practitioner  community.  SMEs  also  stated  that  HSI  work  was 
usually  taken  “out  of  hide”  and  that  HSI  work  is  often  dismissed  or  not 
considered.  For  example,  manpower  authorizations  determined  by 
knowledgeable  practitioners  have  not  been  properly  considered  during  trade-off 
deliberations.  Other  SME  suggestions  included  adding  items  to  the  database 
such  as  “Establishing  a  need  for  HSI  involvement  in  an  IPT”  and  “HSI  planning 
and  management”  elements. 

Another  finding  pertained  to  the  specificity  of  some  database  items.  SMEs 
suggested  that  some  activities  contained  an  excessive  amount  of  details.  This 
may  have  been  true  for  some  activities.  However,  some  HSI  processes  need  to 
be  described  in  detail  so  that  the  nature  of  the  activity  is  truly  understood.  For 
example,  modeling  &  simulation  activities  or  specialized  HSI  tools  may  be 
needed  in  some  types  of  analyses.  If  these  activities  and  tools  are  not  specified, 
the  associated  cost  elements  will  not  be  accurate;  they  may  be  underestimated 
significantly. 

The  SMEs  also  suggested  including  activities  such  as  participation  in 
technical  information  meetings  (in  the  E&MD  phase)  and  contributing  to 
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contractor  taskings.  Contractor  tasking  is  a  controversial  and  obscure  cost  issue. 
Since  contractors  perform  many  of  the  HSI  activities,  contracts  should  include 
indirect  costs  for  these  personnel  as  stated  in  DTM-09-007  (2010). 

Overall,  the  data  collected  from  the  SMEs  supported  the  use  of  the 
proposed  HSI  database  for  AoA  and  Test  &  Evaluation  cost  estimation 
processes.  Although  the  TD  Phase  returned  the  fewest  responses,  it  was  the 
most  informative  for  AoA  guidance.  The  SMEs  elaborated  on  activities  regarding 
AoA  trade-off  studies  and  suggested  assessment  activities  that  were  integrated 
across  HSI  domains.  These  activities  demonstrate  the  need  for  a  significant  level 
of  HSI  practitioner  involvement.  Test  &  Evaluation  activities  were  also  expanded 
to  include  demonstration,  validation  and  verification  processes. 

C.  DATABASE  COLUMNS 

The  database  column  review  resulted  in  many  positive  comments  from  the 
SMEs.  They  agreed  that  all  proposed  data  elements  were  informative  for  HSI 
cost  purposes.  However,  they  recommended  that  the  “Event/Category”  column 
be  refined  by  using  a  cost  tier  structure  to  organize  the  activities.  One  SME 
suggested  that  program  IPTs  tend  to  focus  on  WBS  level  II  and  III  tiers  but  that 
cost  estimation  becomes  increasingly  difficult  at  these  levels  because  of  the 
detail  inherent  in  these  tiers.  It  may  not  be  feasible  to  perform  cost  estimation  for 
HSI  activities  at  the  WBS  tier  III  during  the  AoA.  It  may  be  that  the  tier  III  level  of 
analysis  will  have  to  wait  until  later  in  the  acquisition  process. 

D.  COST  DRIVERS 

The  cost  driver  evaluation  included  a  general  list  of  drivers  identified 
during  the  literature  review.  These  drivers  consisted  of  typical  HSI  domain,  cost 
method  and  System  Engineering  driver  elements.  The  SMEs  reported  that  the 
cost  drivers  were  appropriate  for  an  HSI  cost  database.  Although,  the  cost  driver 
list  was  intended  for  HSI  activities,  SMEs  suggested  that  several  of  the  driver 

elements  could  be  used  in  activities  unrelated  to  HSI  (e.g.,  configuration 
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management,  SLOC).  SMEs  also  classified  the  cost  drivers  as  either  direct  or  in 
direct  and  assigned  cost  drivers  to  each  activity. 

Having  the  SMEs  assign  cost  drivers  to  various  activities  seemed  to  be  a 
viable  approach  to  cost  estimation.  However,  the  SMEs  indicated  that  costs 
rarely  fit  into  such  neat  packages.  In  spite  of  this,  the  SMEs  were  able  to  validate 
the  potential  drivers  for  specific  activities. 

E.  HSI  EFFORT 

The  HSI  Effort  data  provided  the  final  (and  most  interesting)  findings  for 
the  AoA  cost  estimation.  SMEs  provided  HSI  effort  for  overall  phase  to  which 
they  were  assigned,  either  as  a  percentage  of  total  effort  or  as  the  number  of 
man-hours  required.  SMEs  indicated  that  the  most  HSI  activity  should  occur 
during  the  MSA  phase.  SMEs  also  suggested  that  the  HSI  participation  as  a 
whole  (within  the  MSA  phase)  attributed  to  approximately  40-60%  of  effort  (out  of 
100%)  of  a  general  ACAT  I  program.  Within  the  MSA  phase,  HSI  Effort  for 
individual  activities  ranged  from  1%  to  28%.  Those  SMEs  who  expressed  HSI 
participation  in  terms  of  man-hours  provided  some  interesting  insights.  Examples 
of  these  man-hour  estimates  included: 

•  Provide  HSI  Assessment  based  on  best  available  solution 
description  (in  CBA,  AoA,  Study  Guidance,  ICD)  across  all  HSI 
domains  to  define  level  of  risk  in  each:  1000  hours. 

•  Participate  in  AoA  Assessment:  500  hours. 

•  Perform  an  HSI  analysis  of  legacy  systems  with  respect  to  each 
domain  and  address  the  potential  integrated  impact  and  associated 
risks:  500  hours. 

Other  findings  suggested  that  ACAT  I  programs  were  multiyear  efforts,  and 
SMEs  found  it  difficult  to  assess  the  individual  activities  across  multiple  years  and 
multiple  increments. 
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VI.  RECOMMENDATIONS  FOR  FUTURE  WORK  AND 

CONCLUSION 


A.  RECOMMENDATIONS 

The  thesis  explored  HSI  cost  activity  development  for  the  AoA  process  of 
a  generic  ACAT  I  program.  It  also  provides  an  initial  database  that  contains  the 
HSI  activities  that  should  be  included  in  the  cost  estimation  process.  The  thesis 
focused  on  cost  estimation  in  support  of  the  AoA  but  considered  HSI  activities 
that  span  the  entire  acquisition  life  cycle.  Based  on  the  results  and  analysis 
reported  herein,  the  following  recommendations  are  made  for  future  work. 

Identify  the  most  important  HSI  activities  in  each  of  the  oversight 
documents,  including  the  following:  Review  Acquisition  Strategy;  Acquisition 
Plan;  Source  Selection  Plan;  RFP;  Final  Product  Baseline;  Critical  Safety  List 
Items;  PESHE;  and  TEMP. 

•  Perform  HSI  cost  method  comparisons  in  order  to  develop 
variables  for  a  cost  model; 

•  Clarify  HSI  activities  in  the  Acquisition  Management,  Acquisition 
Engineering  and  Support/Sustainability  activity  tiers.; 

•  Organize  the  database  in  terms  of  work  breakdown  structure  cost 
elements; 

•  Identify  the  benefits/impacts  and  risk/cost  relationships  of  HSI 
activities; 

•  Determine  domain  tradeoffs  that  must  occur  to  achieve  accurate 
cost  estimates  or  the  AoA;  and, 

•  Use  a  system  of  interest  to  in  a  case  study  approach  to  verify  the 
database. 
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Unfortunately,  most  programs  do  not  contain  HSI  cost  data.  Therefore,  a 
majority  of  HSI  cost  estimates  will  have  to  rely,  at  least  initially,  on  “rule  of  thumb” 
or  heuristic  approaches.  Two  programs  that  are  making  a  concerted  effort  to 
document  HSI  costs  are  the  MQ-1  and  LCS.  These  programs  should  provide  HSI 
cost  data  that  can  be  used  in  a  parametric  cost  estimation  approach.  The 
parametric  approach  to  estimating  HSI  costs  is  the  ultimate  goal. 

B.  CONCLUSION 

Cost  estimation  is  an  amalgamation  of  systems  that  helps  ensure  the 
validity,  reproducibility  and  affordability  of  any  defense  acquisition  program. 
Capability  assessments  allow  comparisons  with  benchmark  resources  (current  or 
legacy  system  capabilities  and  costs).  The  resultant  cost  data  can  then  be 
applied  to  the  estimation  of  costs  for  future  systems  as  early  in  the  acquisition 
process  as  the  AoA.  This  thesis  was  intended  to  be  general  in  nature  so  that  it 
can  be  applied  to  any  military  service  and  to  any  type  of  system.  The  research 
effort  emphasized  issues  such  as  development/operation  test  and  evaluation 
processes,  program  increments,  operation  and  support,  and  environmental  costs. 
This  method  will  allow  HSI  practitioners  and  other  professionals  the  ability  to 
assess  the  level  of  HSI  effort  required  for  an  acquisition  program  and  to  mitigate 
potential  risks  and  uncertainties  that  may  adversely  impact  the  acquisition  of 
systems.  The  goal  is  to  provide  the  best  possible  system  to  the  Warfighter  that  is 
also  cost  effective  and  timely.  This  thesis  is  an  important  step  in  that  direction. 
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APPENDIX  A.  COMPOSITE  STANDARD  PAY  CHARTS  AND 

REIMBURSEMENT  RATES 


SOLITARY  C  OMPOSITE  ST.ANDARD  PAY  A>'I>  REIMBL’RSEMEN'T  RATES 
DEPARTMEN  T  OF  THE  AIR  FORCE 
FOR  FISC.AL  YEAR  2011 


AN*NT.'AE  RATE 
BILLABLE  TO 


MILITARY 

ANTSL  AL  DOD 

OTHER 

PAY 

A\'ERAGE 

COMPOSITE 

FEDERAL 

OR ADE 

BASIC  PAY 

RATE  - 

AGENCIES 

0-10 

$181,587 

$300,725 

$305,184 

0-9 

181,587 

305,777 

310,236 

0-8 

160.654 

273.032 

277,491 

0-7 

139,237 

246.184 

250,643 

0-6 

115,447 

214.027 

218,486 

0-5 

93,313 

182,049 

186,508 

0-4 

78.866 

160.639 

165,098 

0-3 

63,403 

136,217 

140,676 

0-2 

48,814 

110.344 

114.803 

0-1 

34,673 

87,651 

92,110 

WO-5 

_ 

_ 

_ 

WO-4 

— 

— 

— 

WO-3 

— 

— 

— 

WO-2 

— 

— 

— 

WO-1 

— 

— 

— 

E-9 

$72,291 

$137,410 

$141,869 

E-8 

58,494 

116,749 

121,208 

E-7 

49,755 

104.179 

108.638 

E-6 

40.444 

90.069 

94,528 

E-5 

32,353 

76.538 

80,997 

E-4 

25,667 

63,428 

67,887 

E-3 

21,021 

50,452 

54,911 

E-2 

19,727 

46.434 

50,893 

E-1 

16.9CX) 

41.007 

45,466 

CADETS 

$11,654 

$16,793 

Not  applicable 

«Jotes; 

/  Effective  fiscal  year  2005.  military  personnel  services  for  Foreign  Military  Sales  (FMS)  shall  be  priced  using  the  Composite  Rates 
that  already  include  permanent  change  of  station  (PCS)  expense  and  shall  no  longer  use  the  actual  PCS  expense  for  PCS  moves  to 
siqjport  a  FMS  case.  Effective  F\'  2006.  the  military  personnel  services  for  FMS  shall  be  priced  using  the  DoD  Composite  Rate 
plus  the  acceleration  factor  shov^'n  in  Tab  K-1.  Reimbursement  of  the  acceleration  factor  shall  be  deposited  into  the  E>efense  Health 
Program  (97*0130).  Reimbursement  of  the  per  capita  normal  cost  for  Medicare-eligible  retiree  health  care  (MERHC)  accrual  shall 
be  deposited  into  the  Miscellaneous  Receipts  Account  3041 .  The  next  update  of  the  EX>D  FMR  V'ol-  1 5  Section  070203  will  reflect 
these  change. 

/  The  annual  DoD  composite  rate  includes  the  follovi'ing  military  personnel  appropriation  costs:  average  basic  pay  plus  retired  pay 
accrual.  Medicare-eligible  retiree  health  care  (\1ERHC)  accrual,  basic  allowance  for  housing,  basic  allowance  for  subsistence, 
incentive  and  special  pay,  permanent  change  of  station  expenses,  and  miscellaneous  pay.  Includes  a  per  capita  normal  cost  of 
S5,673  for  MERHC  accrual  —  see  Tab  K-1. 

/  The  annual  rate  billable  to  Other  Federal  Agencies  recovers  additional  military  related  health  care  costs  financed  by  the  Defense 
Health  Program.  The  annual  billable  rate  incltides  an  acceleration  factor  of  S10.132  for  all  personnel.  Excludes  per  capita  normal 
cost  of  S5.673  for  MERHC  accrual  —  see  Tab  K-1 

/  To  confute  a  Daily  Rate,  apply  a  factor  of  .00439.  To  compute  an  Hourly  Rate,  apply  a  factor  of  .00055. 

/  Basic  pay  for  these  officers  is  limited  to  the  rate  of  basic  pay  for  Level  n  of  the  Executive  Schedule,  which  currently  is  $181,587 
per  year. 


Tab  K-5 


Figure  24. 


(From  “Airforce  military  composite  standard  pay  and 
reimbursement  rates,”  2010) 
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MILITARY  C  OMPOSITE  ST.AXDAKD  PAY  .4ND  REIMBLRSEMENT  R.A.TES 


MILIT.ARY 

PAY 

DEP.AJRTMENT  OF  THE 

FOR  FISC.AF  TT.AR  2011  ^ 

.4>rNF.4F  DOD 
.\^^;R.4GF  COMPOSITE 

.AVNT  -AL  RATE 

BILLABLE  TO 

OTHER 

FEDER-4L 

GR 4DE 

P.A^Y 

R.4TE  ■ 

.AGENCIES 

0-10 

$181,587  ’ 

$291,527 

$295,986 

0-9 

181,587 

293,387 

297,846 

0-8 

161.619 

267,571 

272,030 

0-7 

139,396 

238,269 

242,728 

0-6 

117,298 

218.288 

222,747 

0-5 

94.425 

187,883 

192,342 

0-4 

79.869 

169,549 

1 74,008 

0-3 

65,118 

144.700 

149,159 

0-2 

51,126 

113,231 

117,690 

0-1 

37.874 

90.901 

95,360 

WO-5 

$95,112 

$182,380 

$186,839 

WO-4 

82,822 

157.931 

162,390 

WO-3 

70,047 

140.858 

145,317 

WO-2 

57.958 

123,935 

128,394 

WO-1 

— 

— 

— 

E-9 

$71,210 

$140,175 

$144,634 

E-8 

56,707 

119.669 

124,128 

E-7 

47,527 

105,918 

110,377 

E-6 

39.346 

92,856 

97,315 

E-5 

31.585 

79.826 

84,285 

E-4 

25,354 

67,296 

71,755 

E-3 

21,516 

55,557 

60.016 

E-2 

19,676 

50.071 

54,530 

E-1 

17,122 

44.599 

49,058 

CADETS 

$11,648 

$17,241 

Not  applicable 

Notes; 

1/  Effective  fiscal  year  2005.  imlitar>'  personnel  services  for  Foreign  Military  Sales  (FMS)  shall  be  priced  using  the  Composite  Rates 
that  already  include  permanent  change  of  station  (PCS)  expense  and  shall  no  longer  use  the  actual  PCS  expense  for  PCS  moves  to 
support  a  FMS  case.  Effective  FY  2006.  the  military  personnel  services  for  FMS  shall  be  priced  using  the  E>oD  Composite  Rate 
plus  die  acceleration  factor  shown  in  Tab  K-1  Reimbursement  of  the  acceleration  factor  shall  be  deposited  into  the  Defense  Health 
Program  (97*0130).  Reimbursement  of  the  per  capita  normal  cost  for  Medicare-eligible  retiree  health  care  (\IERHC)  accrual  shall 
be  deposited  into  the  Miscellaneous  Receipts  Account  3041 .  The  next  update  of  die  DoD  FMR  Vol.  1 5  Section  070203  will  reflect 
these  change. 

1!  The  annual  DoD  composite  rate  includes  the  following  miUtary  personnel  appropriation  costs:  average  basic  pay  plus  retired  pay 
accrual.  Medicare-eligible  retiree  health  care  (MERHC)  accrual,  basic  allowance  for  housing,  basic  allowance  for  subsistence, 
mcentive  and  special  pay.  permanent  change  of  station  expenses,  and  miscellaneous  pay.  Includes  a  per  capita  normal  cost  of 
S5.673  for  MERHC  accrual  -  see  Tab  K-1. 

3/  The  annual  rate  billable  to  Other  Federal  Agencies  recovers  additional  military  related  healdi  care  costs  financed  by  die  Defense 
Health  Program.  The  annual  billable  rate  includes  an  acceleration  factor  of  $10,132  for  all  personnel.  Excludes  per  capita  normal 
cost  of  S5.673  for  MERHC  accrual  —  see  Tab  K-1. 

4/  To  compute  a  Daily  Rate,  apply  a  factor  of  .00439.  To  compute  an  Hourly  Rate,  apply  a  fector  of  .00055. 

5/  Basic  pay  for  these  officers  is  limited  to  the  rate  of  basic  pay  for  Level  n  of  the  Executive  Schedule,  which  currendy  is  $181,587 
per  year. 


Tab  K-3 

Figure  25.  (From  “Navy  military  composite  standard  pay  and  reimbursement 

rates,”  2010) 
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APPENDIX  B.  HEALTH  HAZARD  COST 


Related  cost 
component 

Equation 

variable 

Variable 

value 

Description 

All  (except 
death  costs) 

P. 

See  Table  1 

Hazard  Probability  (HP)  -  Probability  of 
exposure  per  year,  based  on  the  determined 
hazard  probability  category 

All  (except 
death  costs) 

Sk 

See  Table  1 

Hazard  Severity  (HS)  factor  based  on  the 
determined  hazard  severity  category 

All  (except 
death  costs) 

Ns 

No.  of  systems 

Number  of  systems,  the  total  number  individual 
items  of  materiel,  equipment,  or  weapon 
systems  in  Army  inventory 

All  (except 
death  costs) 

Np. 

No.  of  persons 

Number  of  persons  per  system,  or  crew  size 
for  system,  or  item 

All  (except 
death  costs) 

N. 

Calculated 

Total  number  of  people  exposed  to  hazard  per 
year  for  the  systems  or  items 

Clinic  costs 

c. 

Calculated 

Cost  of  clinic  visits 

Clinic  costs 

N, 

Calculated 

Number  of  people  injured  or  ill 

Clinic  costs 

N. 

Calculated 

Number  of  clinic  visits 

Clinic  costs 

V, 

0.75 

Visit  constant  as  result  of  exposure.  The  visit 
constant  (Ve)  equals  0.75  and  is  based  on  expo¬ 
sure  to  a  health  hazard  that  results  in  illness  or 
injury.  We  assumed  that  if  an  exposure  event 
occurs,  then  75  percent  of  all  persons  exposed 
to  the  hazard  will  visit  the  clinic  for  an  examina¬ 
tion  to  determine  whether  any  injury  has 
occurred. 

Clinic  costs 

See  Table  5 

Incidence  of  injury  or  illness  based  on  the 
determined  risk  level  for  the  individual  item  of 
materiel 

Clinic  costs 

Nc 

See  Table  6 

Number  of  visits  by  injured  or  ill  personnel  based 
on  the  determined  hazard  severity  category.  The 
hazard  severity  category  determines  the  serious¬ 
ness  of  the  medical  outcomes  that  could  occur. 
As  the  severity  increases,  the  number  of  clinic 
visits  increases.  For  this  cost  component,  based 
on  values  selected  by  a  panel  of  experts,  we 
assigned  the  number  of  visits  based  on  the 
hazard  severity  category  and  the  potential 
medical  outcomes. 

Clinic  costs 

Pc 

$122  per  visit 

Average  fee  per  clinic  visit,  based  on  the 
average  of  various  types  of  clinic  service  visit 
fees.  We  found  the  average  fee  was  S122  per 
clinic  visit. 

Figure  26.  Equation  variables  (From  "Estimating  the  health  hazard  costs,” 

1998) 
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Hospitalization 

costs 

c. 

Calculated 

Cost  of  hospitalization 

Hospitalization 

costs 

Calculated 

Number  of  persons  hospitalized 

Hospitalization 

costs 

N. 

Calculated 

Number  of  hospital  days 

Hospitalization 

costs 

See  Table  7 

Incidence  of  hospitalization  based  on  the 
determined  risk  level  for  the  individual  item  of 
materiel 

Hospitalization 

costs 

See  Table  8 

Factor  for  the  average  number  of  days  in 
hospital  per  person  based  on  historical  hospital 
stay  distribution 

Hospitalization 

costs 

See  Table  9 

Factor  for  the  hospitalization  population 
distribution  for  average  number  of  days  in 
hospital 

Hospitalization 

costs 

Fh 

$1 ,669  per  day 

Average  fee  per  hospital  day.  Average  cost  based 
on  various  types  of  hospital  diagnosis-related 
groups  and  the  classification  of  the  disease.  We 
found  the  average  hospital  fee  was  Si  ,669  per 
day. 

Lost  time  costs 

c, 

Calculated 

Cost  of  days  of  lost  time 

Lost  time  costs 

Np, 

Calculated 

Number  of  persons  losing  time 

Lost  time  costs 

N, 

Calculated 

Number  of  lost  workdays 

Lost  time  costs 

See  Table  10 

Incidence  of  lost  time  based  on  the  determined 
risk  level  for  the  individual  materiel  item 

Lost  time  costs 

See  Table  1 1 

Factor  for  the  number  of  lost  workdays  per 
person  based  on  historical  lost  workday 
distribution 

Lost  time  costs 

See  Table  12 

Lost  time  population  distribution  based  on 
average  lost  workday  distribution 

Lost  time  costs 

w. 

$53.97  per  day 

Average  wage  per  day.  We  based  the  average 
wage  per  day  (Wd)  on  the  salaries  and  numbers 
of  persons  drawing  that  salary  for  a  selected 
group  of  personnel.  We  determined  an  average 
wage  to  be  $53.97  per  day. 

Lost  time  costs 

B, 

1.41 

Wage  fringe  benefit  factor.  We  assigned  the  fringe 
benefit  factor  (Bf)a  value  of  1.41.  It  is  a  standard 
factor  within  the  government  used  for  program¬ 
ming  personnel  budget  requirements  and  is 
representative  of  other  corporate  benefit  factors. 

Disability  costs 

Calculated 

Cost  of  disabilities 

Figure  27.  Equation  variables  continued  (From  “Estimating  the  health  hazard 

costs,”  1998) 
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Disability  costs 

pd 

Calculated 

Number  of  persons  disabled 

Disability  costs 

1, 

See  Table  13 

Incidence  of  VA  disability  based  on  the  deter¬ 
mined  risk  level  for  the  individual  item  of 
materiel,  equipment,  or  weapon  system 

Disability  costs 

T, 

0.25 

VA  disability  adjustment  factor  for  delayed 
disability  (5  years/20  years) 

Disability  costs 

D, 

See  Table  14 

VA  disability  population  factor  based  on 
historical  rate  of  disability  distribution 

Disability  costs 

B. 

See  Table  15 

VA  disability  compensation  factor  per  month 
per  rate  of  disability 

Disability  costs 

0.001 

Incidence  of  active-duty  temporary  disability 
(1  case/1000  persons) 

Disability  costs 

B. 

$9,242  per 
person 

Active-duty  temporary  disability  compensation 
per  year 

Disability  costs 

'p 

0.011 

Incidence  of  active-duty  permanent  disability 
(11  cases/1000  persons) 

Disability  costs 

Bp 

$12,864  per 
person 

Active  duty  permanent  disability  compensation 
per  year 

Rehabilitation 

costs 

c, 

Calculated 

Cost  of  rehabilitation 

Rehabilitation 

costs 

N, 

Calculated 

Number  of  rehabilitation  cases 

Rehabilitation 

costs 

D 

r 

See  Table  16 

Eligible  VA  disability  population  factor  based  on 
rate  of  disability  distribution  equal  to  or  greater 
than  20  percent 

Rehabilitation 

costs 

Qr 

0.05 

VA  rehabilitation  qualification  factor 
(5  cases/100  persons  eligible) 

Rehabilitation 

costs 

B, 

$12,000  per 
year  per  person 

VA  rehabilitation  benefit  per  year  per  person.  We 
estimated  to  be  $12,000  per  year  per  person. 
Rehabilitation  benefits  may  vary  per  person,  but 
we  considered  $12,000  to  be  a  reasonable 
estimate.  Other  benefits  may  be  available  for 
eligible  disabled  persons,  but  we  did  not  consider 
these  other  benefits. 

Death  costs 

Calculated 

Cost  of  death 

Death  costs 

N.. 

See  Table  17 

Number  of  deaths  per  year 

Death  costs 

B*. 

$200,000 

Death  benefit  and  expenses 

Figure  28.  Equation  variables  continued  (From  “Estimating  the  health  hazard 

costs,”  1998) 
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Table  5.  Incidence  of  Illness  or  Injury  (I,)  for  System  Risk  Categories 


System  risk  category 

Incidence  rate  (I,) 

High 

0.122 

Medium 

0.095 

Low 

0.067 

Table  6.  Number  of  Clinic  Visits  (NJ  for  Hazard  Severity  Categories 


Hazard  severity  category 

Number  of  clinic  visits  (NJ 

1 

5 

II 

3 

III 

2 

IV 

1 

Table  7.  Incidence  of  Hospitalization  (1^)  for  System  Risk  Categories 

System  risk  category 

Hospitalization  rate  (1,^) 

High 

0.013 

Medium 

0.007 

Low 

0.0005 

Table  8. 

Factors  for  Average  Number  of  Days  in  Hospital  (D^_,)  (days/person) 

Length  of  stay  in  hospital 

Factor (D„J 

<2  days 

1.0 

2-5  days 

3.5 

6-30  days 

18.0 

>30  days 

30.0 

Table  9.  Factors  for  Hospitalization  Population  Distribution  (D^^) 
by  Length  of  Stay  in  Hospital  for  System  Risk  Categories 


System  risk  category 

Length  of  stay  in  hospital 

<2  days 

2-5  days 

6—30  days 

>30  days 

High 

0.40 

0.35 

0.17 

0.08 

Medium 

0.40 

0.36 

0.18 

0.06 

Low 

0.42 

0.37 

0.20 

0.02 

Figure  29.  Health  Hazard  Cost  Method  Criteria  (From  “Estimating  the  health 

hazard  costs,”  1998) 


124 


Table  10.  Incidence  of  Lost  Time  (I,)  for  System  Risk  Categories 


System  risk  category 

Lost  time  rate  (1,) 

High 

0.055 

Medium 

0.054 

Low 

0.028 

Table  11. 

Factors  for  Average  Number  of  Days  of  Lost  Time  (0,^)  (days/persen) 


Number  days  of  lost  time 

Factor  (D,^) 

<2  days 

1.0 

2-5  days 

3.5 

6-30  days 

18.0 

>30  days 

30.0 

Table  12.  Factors  for  Lost  Time  Population  Distribution  (D„) 
by  Days  of  Lost  Time  for  System  Risk  Categories 


System  risk  category 

Lost  time 

<2  days 

2-5  days 

6-30  days 

>30  days 

High 

0.22 

0.30 

0.29 

0.20 

Medium 

0.20 

0.33 

0.31 

0.16 

Low 

0.15 

0.43 

0.38 

0.04 

Table  13.  Incidence  of  VA  Disability  (I,)  for  System  Risk  Categories 


System  risk  category 

VA  disability  factor  (IJ 

High 

0.032 

Medium 

0.012 

Low 

0.00005 

Table  14.  Factors  for  Disability  Population  Distribution  (D^) 
by  Degree  of  Disability  for  System  Risk  Categories 


System  risk  category 

Degree  of  disability 

10% 

20%-50% 

60%-90% 

100% 

High 

0.44 

0.42 

0.10 

0.04 

Medium 

0.44 

0.44 

0.09 

0.03 

Low 

0.43 

0.48 

0.08 

0.01 

Figure  30.  Health  Hazard  Cost  Method  Criteria  (From  “Estimating  the  health 

hazard  cost,”  1998) 
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Tabl«^  1£.  VA  Disability  Cempenwti^n  Facten  |D  J  by  Degre# 
of  Di&abiliry  {d^lori/iuenth/p^rsenf 


Degree  disabitity 

VAdisabiniy  comperrsatian  faciter  |B  J 

1(K-. 

sauDO 

2nf^Ar-sm-, 

I34.D.25 

3S1E.50 

S1.B65.D0 

Table  16.  Eligiible  VA  Disability  Papulation  Distribution  Factori  |DJ 
by  Degree  of  Disability  for  System  Risk  Categories 


System  risk  calegory 

tQ% 

Degree  uf  disabilhy 
2S!fV.r-SD^  eO'^i.-BO’^t 

too^ 

High 

OJO 

0.42 

Q.tQ 

0.04 

Medium 

0.0 

0.44. 

0.09 

0.03 

Low 

0.0 

0.4S 

o.oa 

O.Dt 

Table  IT.  Number  of  Deaths  (N^^l  for  Hazard  Severity  Caregeriei 


Hazard  severity  category 

HumberoF  deaths 

r 

1 

II 

0 

III 

0 

IV 

0 

Table  IS.  Health  Hazards  and  Associated  Risk  Indites  for  System  X 


Hazard  cntcigory 

Ttiab:  htozDid 

as9e-»ment  acMefity 
Hazard  codeCHAC^  ode^ory 

Hozend 

probafcSty 

ChcfnicoJ  substaiHm. 

Wecfiom  combiBlian  pratdijcia 

t 

1 

A 

Chemioal  nubiUuMm. 

Ftre  nlimguislwg  n^enla 

£ 

II 

C 

ChcrnidoJ  substaiHm. 

Ccrbnn  [fitKide 

3 

■  1 

□ 

dcousticBJ  energy 

Impulse  noise 

£ 

II 

c 

Ac^sti^d  energy 

SteDdy-slaXe-  noise 

£ 

II 

c 

Tempemture  Kiretnee 

Coid  stress 

£ 

■ 

c 

TctnperntiMe  siremefl 

Hest  sire-ss 

£ 

II 

c 

Oiygen  deiioiemy 

Cbrygen  deficKney  (ventilation) 

£ 

II 

c 

Radiation  energy 

'Nornnizinig  racfiatiDn 

£ 

II 

c 

Rasiatian  energy 

Ionizing  rw&liiDn 

4 

II 

E 

Figure  31.  Health  Hazard  Cost  Method  Criteria  (From  “Estimating  the  health 

hazard  cost,”  1998) 
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APPENDIX  C.  LIFE  CYCLE  PHASE  HSI  ACTIVITIES 


Event 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

A. 

Provide  FISI  activity  input 
during  Capabilities  Based 
Assessment  for  ICD 

*DOTMLPF  Analysis, 
Analysis  of  Materiel  and 
Non-Materiel,  and  lessons 

Pre-Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Documentation  to  match  life 
cycle  needs  (Breadth  and 
depth  of  required 

B. 

Provide  HSI  input  into 

HSIP  for  the  System 
Engineering  Plan  (SEP) 

*Appoint  HSI  Lead. 

**Work  with  PM  to 
determine  if  standalone 

Pre-Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Complexity,  Maintenance 
Concept,  Manning,  Mission, 
Operational  Capability, 

C. 

Provide  a  total  system 
analysis  based  on 
functional  relationship 

^Analyze  and  assess  trade 
space,  related 
constraints/requirements 

Pre-Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Complexity,  Maintenance 
Concept,  Manning,  Mission, 
Operational  Capability, 

D 

Provide  HSI  considerations 
into  AoA  Study  plan 

*ldentify  AoA  manpower 
estimating  tools,  analysis 
techniques. 

Pre-Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

General  Drivers: 

Complexity,  Maintenance 
Concept,  Manning,  Mission, 

E. 

Provide  input  into  ESOH 
integration  Strategy. 

*Define  HSI  roles  and 
activities  for  Environmental, 
Safety  and  Occupational 

Pre-Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Complexity,  Maintenance 
Concept,  Manning,  Mission, 
Operational  Capability, 

F. 

Provide  a  HSI  assessment 
based  on  best  available 
solution  description  (in 

*lnfrom  Sustainment  and 
affect  design  the  design 
and  support  plans  that 

Pre-Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Manning 

E. 

Perform  a  HSI  analysis  of 
legacy  systems  with  respect 
to  each  domain  and 

*Review  the  legacy 
system's  Post 
Implementation  Review 

Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Cognitive  Workload, 
Complexity,  Initial  Training 
(Instructor  &  Key 

F. 

Contribute  to  assessment 
and  development  of  system 
concepts  /Materiel 

*Consider  personnel 
alternatives  based  on  AoA. 
**Determine  appropriate 

Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Initial  Training  (Instructor  & 
Key  Personnel),  Initial 
Training  (New  Equipment  at 

G. 

Assist  in  development  of 
cost  estimate  based  on 
anticipated  HFE, 

‘Provide  input  to  Manpower 
Estimate  Reports 

Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Initial  Training  (Instructor  & 
Key  Personnel), 

Maintenance  Requirement, 

H. 

Conduct  Mission  Task 
Analysis. 

‘Define  the  human 
performance  characteristics 
of  the  user  population. 

Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Cognitive  Workload, 
Complexity,  Criticality  of 
Task,  Difficulty  of  Task, 

Figure  32.  Materiel  Solution  Analysis  HSI  Activities 
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Event 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

I. 

Conduct  Initial  Top  Down 
Requirements  Analysis. 

*Participate  in  an  IPT  and 
provide  preliminary  input 
into  Manpower  and 

Materiel 

Solution 

Analysis 

Manpower 

Cognitive  Workload, 
Frequency  of  Task, 

Manning,  Mission, 

J. 

Assist  in  performance  of 
Manpower  /Personnel 
alternatives  and  concept 

*Use  findings  from  mission 
task  analysis  and  other 
personnel  assessments. 

Materiel 

Solution 

Analysis 

Personnel, 

Manpower 

Operational  Capability, 
Personnel 

experience/continuity 

K. 

Identify  manning  needs, 
manpower  lessons  learned, 
and  high  drivers  from 

*Establish  preliminary 
manpower  thresholds  and 
objectives. 

Materiel 

Solution 

Analysis 

Manpower, 

Personnel 

Complexity,  Maintenance 
Requirement,  Manning, 
Mission,  Operational 

L. 

Assist  in  linear  modeling 
and  studies  analysis  of 
system  approach 

*Assist  in  Effectiveness 
Analysis  Working  Group 
(EAWG). 

Materiel 

Solution 

Analysis 

Manpower,  HFE, 
Training 

Manning, 

Modeling&Simulation, 
Operational  Capability, 

M. 

Conduct 

Manpower/T  raining 
assessments  to  determine 

*Conduct  preliminary 
learning  performance  and 
solution  analysis  to  justify 

Materiel 

Solution 

Analysis 

Training, 

Personnel 

Cognitive  Workload,  Initial 
Training  (Instructor  &  Key 
Personnel),  Initial  Training 

N. 

Assist  in  deriving 

Systematic  Prediction  of 
Conditions  through  Initial 

*Review  draft  /final 
assessments  and 
associated  costs. 

Materiel 

Solution 

Analysis 

ESOH 

Criticality  of  Task, 

Frequency  of  Task, 

Manning,  Mission,  Physical 

O. 

Participate  in  AoA 
assessments. 

*Optimize  MPT  for  AoA. 
**ldentify  capability  gaps  in 
Manpower  reduction. 

Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Cognitive  Workload, 
Complexity,  Maintenance 
Requirement,  Manning, 

P. 

Define  concept  capabilities 
and  requirements. 

*Determine  HFE  and 

Human  performance 
requirements. 

Materiel 

Solution 

Analysis 

HFE,  Manpower, 
Personnel 

Cognitive  Workload, 
Complexity,  Frequency  of 
Task,  Maintenance 

Q. 

Provide  HSI  input  into 
Technology  Development 
Strategy. 

*Provide  preliminary  inputs 
into  MDA  cost  estimates  for 
TD  phase  to  support  the 

Materiel 

Solution 

Analysis 

HFE,  Manpower, 

Personnel, 

Training 

Cognitive  Workload, 
Complexity,  Criticality  of 
Task,  Difficulty  of  Task, 

R. 

Provide  HSI  Input  into  Life- 
cycle  Sustainment  Plan. 

*Provide  preliminary 
considerations  into 
Habitability,  Training,  and 

Materiel 

Solution 

Analysis 

All  Domains 

(Integrated 

Assessments) 

Demilitarization  and 
Disposal,  Migration 
complexity  (Influence  of 

S. 

Provide  HSI  input  into 
Support  System 

Engineering  Technical 

*Provide  input  into  the  Initial 
Technical  Review  (ITR)  and 
Alternative  System  Review 

Materiel 

Solution 

Analysis 

HFE,  Manpower, 
Training 

Complexity,  Manning, 
Mission,  Requirements 
understanding  (Subjective 

Figure  33.  Materiel  Solution  Analysis  HSI  Activities  (continued) 
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EVENT 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

GUIDE 

Review  all  the  inputs  to 
the  Technology 
Development  Phase 

1.0  Update  HSI 
domain  effects. 

1.1  Review  and  update 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  tools, 
simulation,  software, 
analyses,  training. 

A. 

Determine  the  need  for 
an  HSI  IPT  or  a 

Working  Group. 

*Review  outputs  from 
user  CONOPs 
developments,  needs 

Technology 

Development 

All  as 

appropriate  to 
the  solution 

Manning,  tools, 
simulation,  software, 
analyses,  training. 

B. 

Establish  a  HSI  IPT  or 
Working  Group  - 
consistent  with  the  size 

*ldentify  candidate 
participants  /  team 
members  to  participate 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  Personnel 

experience/continuity 

(Subjective 

GUIDE 

Throughout  the  next  5 
steps  C-G  conduct 
trade  offs  and  support 

*  Participate  in  AoA  to 
ensure  that  HSI 
considerations  have 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

C. 

Determine  all  relevant 
critical  operational 
issues  and  determine 

*  Interpret  User  Needs, 
Analyze  Operational 
Capabilities  & 

Technology 

Development 

HFE, 

Manpower, 

Training, 

Maintenance  Concept, 
Maintenance 
Requirement,  Mission, 

D. 

Callout 

Develop  system 

performance 

(&constraints) 

*  Ensure  HSI  criteria 
are  traceable  back  to 
defined  system 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  travel. 

E. 

Callout 

Develop  functional 
definitions  for 
Enabling/Critical 

*  Define  HSI  criteria  for 
weapon  system, 
support,  equipment. 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  tools, 
evaluations,  analyses. 

F. 

Callout 

Decompose  functional 
definitions  into  Critical 
Component  Definition 

*  Update  system  HSI 
criteria. 

*  Assess  HSI  impacts 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  simulation, 
tools,  evaluations. 

G. 

Identify  HSI  technical 
risks  and  mitigation 
strategies  for 

*ldentify  detailed 
administrative 
processes  for  HSI 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Difficulty  of  Task, 
Maintenance 
Requirement,  Physical 

Figure  34.  Technology  Development  Phase  HSI  Activities 
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EVENT 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

GUIDE 

Throughout  the  next  7 
Steps  H-O  conduct 
trades  that  are  spread 

*  Conduct  trade 
studies  on  threshold 
and  objective  levels  of 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

H. 

Demonstrate 

enabling/critical 

technology 

*  Integrate  evaluations 
of  critical  technologies 
across  all  functional 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Maintenance  Concept, 
Manning,  Mission, 
Operational  Capability, 

I. 

Provide  integrated  HSI 
assessment 
methodology  for 
technology  down 

*Define  relevant 
criteria  across  the  HSI 
domains  and  for 
overall  integration 

Technology 

Development 

Human  Factors 

Engineering 

(HFE) 

Complexity,  Manning, 
Modeling&Simulation, 
Testing  Type- 
campaign  or  force- 

J. 

Refine  and  update 
requirements. 

*Update  HFE  for  HMI, 
HCI,  Workspace, 
Human  Performance 

Technology 

Development 

HFE, 

Manpower, 

Personnel, 

Manning 

K. 

Assist  in  technology 
selection. 

*Address  all  relevant 
requirements  that  have 
HSI  implications. 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  Specialty 
Personnel,  Training 
Analysis  and  Design 

L. 

Perform  analysis  and 
recommendations  for 
system  technology 

*Perform  Operational 
Workflow  Analysis. 
*Perform  Manpower, 

Technology 

Development 

HFE, 

Manpower, 

Personnel, 

Cognitive  Workload, 
Criticality  of  Task, 
Difficulty  of  Task, 

M. 

Assist  in  development 
of  System  Functional 
Specs  &  Verification 

*Perform  HSI 

Functional  Allocation 
Analysis. 

Technology 

Development 

HFE, 

Manpower, 

Personnel, 

Cognitive  Workload, 
Manning,  Mission, 
Operational  Capability, 

N. 

Participation  in 
Integrated  Test  Team 
(ITT)  and  evaluation  or 

*Determine  scope  of 
Safety  and 

Occupational  Health 

Technology 

Development 

Environment, 

HFE, 

Manpower, 

Complexity,  Manning, 
Modeling&Simulation, 
Testing  Type- 

O. 

Develop  T&E  /  V&V 
program  plans 
inclusive  of  all  the 

*Develop  HSI 

Methods,  Metrics, 
Analysis  and  Tools. 

Technology 

Development 

HFE 

Complexity,  Manning, 
Mission, 

Modeling&Simulation, 

Figure  35.  Technology  Development  Phase  HSI  Activities  (continued) 
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EVENT 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

Guide 

Throughout  the  next  7 
Steps  P-S  conduct 
trades  that  are  spread 

*  Conduct  trade 
studies  on  threshold 
and  objective  levels  of 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  modeling  / 
simulation,  analyses, 
trade  studies. 

P. 

Callout 

Demonstrate 

enabling/critical 

technology 

*  Integrate  evaluations 
of  critical  technologies 
across  all  functional 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  travel,  tools. 

Q. 

Callout 

Demonstrate  system  & 
prototype  functionality 
versus  plan. 

*  Evaluate  critical 
technologies  from  an 
HSI  perspective 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning 

R. 

Callout 

Demo/Model 
integrated  system 
versus  performance 

*  Evaluate  critical 
technologies  from  an 
HSI  perspective. 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  models  / 
simulation 

S. 

Callout 

Demonstrate  & 
validate  System  & 
Technology  Maturity 

*  Ensure  applicable 

HSI  elements  are 
embedded  in  the 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning 

GUIDE 

Insure  the  production 
and  completion  of 
inputs  to  the  following 

1.0  Incorporate  domain 
considerations  into 
baseline  parameters 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  tools, 
simulation,  software, 
analyses,  training. 

T. 

Review  and  provide 
input  into  Preliminary 
design  and  allocation 

*Provide  HSI  inputs 
into  all  relevant  areas 
(e.g.  systems  safety. 

Technology 

Development 

Environment, 

HFE, 

Occupational 

Manning,  tools, 
simulation,  software, 
analyses,  training. 

U. 

Provide  HSI  input  into 
the  Capability 
Development 

*Provide  design 
considerations  for 
environmental  facts. 

Technology 

Development 

Environment, 

HFE, 

Manpower, 

Manning,  Mission 

V. 

Provide  input  into  the 
Acquisition  Strategy  for 
goals,  objectives  and 

*Provide  an  outline 
methodology  to: 
**Determine 

Technology 

Development 

HFE, 

Manpower, 

Personnel, 

Manning,  Mission, 
Operational  Capability 

Figure  36.  Technology  Development  Phase  HSI  Activities  (continued) 
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EVENT 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

W. 

Provide  input  into  Acquisition  Program 
Baseline  (APBs). 

‘Provide  input  for  KPP,  KSA,  Threshold  and  Objective  values 
and  requirements  considerations. 

‘Operationally  define  performance  parameters,  measures  of 

Technology 

Development 

HFE, 

Manpower, 

Personnel 

Level  of  service  requirements 
(Subjective  difficulty  of  satisfying  the  key 
performance  Parameters),  Manning, 

X. 

Incorporate  requirement  for  contractor 
generated  HSIP(s)  and  associated 
source  selection  criteria  in  Request  for 
Proposals  (RFPs). 

‘Review  RFP  language,  CDRLs  and  DIDs  for  HSI  Planning 
activities  for  each  human  performance. 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning 

Y. 

Conduct  formal  development  of 
human-centered  source  selection 
criteria  for  RFPs  at  Milestone  (MS)  A. 

‘Conduct  HSI  review  of  source  selection  criteria  for 
developmental  RFPs. 

“Review  all  source  selection  criteria  for  HSI  relevance. 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Complexity,  Maintenance  Requirement, 
Manning,  Mission,  Platforms,  Specialty 
Personnel 

Z. 

Apply  analysis  in  findings  to  Test  and 
Evaluation  Strategy  (TES),  LCMP  and 
TEMP 

‘Conduct  Special  analysis  to  update  thresholds,  objectives, 
and  evolving  criteria  for  DT&E  and  eventually  OT&E. 

‘Update  HSI  test  plans  as  necessary 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning 

AA. 

Provide  input  into  System  Engineering 
Review  Process  (SRR). 

‘Provide  input  into  SRR  for  implication  of  humans  into  system. 
SFR  for  operation  level  functional  requirements  as  IPT 
member. 

Technology 

Development 

HFE, 

Manpower, 

Survivability 

Complexity,  Maintenance  Requirement, 
Manning,  Mission,  Operational 

Capability,  Personnel/team  capability 

BB 

Provide  input  into  System  Engineering 
Review  Process  (SFR) 

‘  Address  HSI  requirements  in  the  system  functional  baseline 
and  in  conjunction  with  the  lower-level  performance 
requirements. 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning 

CC 

Provide  input  into  System  Engineering 
Review  Process  (PDR) 

‘  Ensure  domain-specific  performance  requirements  are 
included  in  the  preliminary  design. 

‘  Review  subsystem  requirements  to  address  HSI  issues 

Technology 

Development 

All  Domains 

(integrated 

assessment) 

Manning,  Travel 

Figure  37.  Technology  Development  Phase  HSI  Activities  (continued) 
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Event 

HSI_Activity 

HSI_Sub  Activity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

A. 

Provide  HSI  input  to 
the  TEMP. 

*lncorporate  use  of 
HSI  IPT  (Evaluation 
Plans);  Identify 

E&MD:  Integrated 
System  Design 

All  Domains 

(Integrated 

Assessment) 

Complexity, 

Manning, 

Mission, 

B. 

Finalize  HSI 
requirements  by 
obtaining 

*Determine 

Assessment 

methods; 

E&MD:  Integrated 
System  Design 

All  Domains 

(Integrated 

Assessment) 

Complexity, 
Manning,  Mission, 
Model  ing&Simulati 

C. 

Assist  in 

development  of 
detailed  design 

*Provide  guidance 
for  supporting 
development  of 

E&MD:  Integrated 
System  Design 

All  Domains 

(Integrated 

Assessment) 

Complexity, 
Frequency  of  Task, 
Mission, 

D. 

Assess  detailed 
system  designs  to 
ensure  that  they 

^Validate 
manpower 
estimate  basis 

E&MD:  Integrated 
System  Design 

All  Domains 

(Integrated 

Assessment) 

Manning,  Mission, 

Operational 

Capability, 

E. 

Participate  in 

Prototypes/Test 

Materials 

Interactively 
develop  and  review 
prototypes. 

E&MD:  Integrated 
System  Design 

Human  Factors 
Engineering 

Cognitive 

Workload, 

Complexity, 

F. 

Evaluate  Prototypes 
(or  Test  Materials) 
for  HSI  criteria  . 

Perform  heuristic 

evaluations. 

Cognitive 

E&MD:  Integrated 
System  Design 

All  Domains 

(Integrated 

Assessment) 

Cognitive 

Workload, 

Complexity, 

G. 

Participate  in  design 
demonstrations  (to 
include 

*Verify  detailed 
system  designs. 
*Evaluate  initial 

E&MD:  Integrated 
System  Design 

All  Domains 

(Integrated 

Assessment) 

Mission, 

Operational 

Capability, 

H. 

Evaluate  HSI  criteria 
for  the  system 
during  component  / 

*Complete  data 
collection  during 
DT&E  and  OT&E  for 

E&MD:  Integrated 
System  Design 

All  Domains 

(Integrated 

Assessment) 

Complexity, 
Platforms,  Testing 
Type-campaign  or 

I. 

Perform  HSI 

evaluation  on 
system 

^Determine  impact 
on  human 
performance  and 

E&M  D:System 
Capability  & 
Manufacturing 

All  Domains 

(Integrated 

Assessment) 

Complexity, 
Modeling&Simulati 
on.  Personnel/team 

Figure  38.  Engineering  and  Manufacturing  Development  Phase  HSI  Activities 
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Event 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

J. 

Assist  in 

Development  of 

Test  Plans  for 

^Develop  test  plans 
for  human 
performance 

E&MD:System 
Capability  & 
Manufacturing 

Human 

Factors 

Engineering 

Occupational 
Training,  Testing 
Type-campaign  or 

K. 

Participate  in  HSI 
Test  events. 

*Perform  analysis 
and  reporting 
functions  related  to 

E&MD:System 
Capability  & 
Manufacturing 

All  Domains 

(Integrated 

Assessment 

Modeling&Simulati 
on.  Operational 
Capability,  Testing 

L. 

Participate  in  ECP 
development. 

^Generate  and 

review  HFE  related 

ECPs  for  human 

E&MD:System 
Capability  & 
Manufacturing 

Human 

Factors 

Engineering 

Manning 

M. 

Provide  HSI  updates 
to  Acquisition 
Strategy  and  APB. 

^Ensure  continued 
emphasis  on  HSI 
requirements  met 

E&MD:System 
Capability  & 
Manufacturing 

Manpower 

Personnel, 

Training 

Manning 

N. 

Provide  HSI  input 
into  Capability 
Production 

^Provide  input  to 
CPD  based  on 

lessons  learned 

E&MD:System 
Capability  & 
Manufacturing 

Environme 

nt. 

Habitability 

Manning 

O. 

Provide  HSI  input 
into  Technical 

Reviews  and 

^Contribute  HSI- 
related  entry  and 
exit  criteria  for 

E&MD:System 
Capability  & 
Manufacturing 

Human 

Factors 

Engineering 

Manning, 

Operational 

Capability, 

P. 

Provide  HSI  input 
for  Error  and  Fault 
analysis. 

*Examine  workload 
levels  at  micro-, 
meso-,  and  macro- 

E&MD:System 
Capability  & 
Manufacturing 

Human 

Factors 

Engineering 

Complexity, 
Criticality  of  Task, 
Difficulty  of  Task, 

Figure  39.  Engineering  and  Manufacturing  Development  Phase  HSI  Activities  (continued) 
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Event 

HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

A. 

HSI  Planning  and 

Management 

*As  member  of  Integrated 
Product  Team,  review,  update 
and  trade-off  prior 

Production 

and 

Deployment 

All  Domains 
(Integrated) 

Manning,  Requirements 
understanding.  Level  of 

Service  Requirements,  Time 

B. 

Review  and  develop  HFE 
installation  Criteria. 

‘Consider  human  performance 
workload  and  safety. 

‘Identify  gaps  or  potential 

Production 

and 

Deployment 

HFE 

Complexity,  Level  of  service 
requirements  (Subjective 
difficulty  of  satisfying  the  key 

C. 

Verify  HSI  requirements  for 
installation  criteria. 

‘Assist  in  HSI  verification  of 
system  installation 
requirements 

Production 

and 

Deployment 

All  Domains 
(Integrated) 

Cognitive  Workload,  Level  of 
service  requirements 
(Subjective  difficulty  of 

D. 

Assist  in  T&E  HSI  analyses  of 
system  design  and  installation 
(including  LRIP  deficiencies). 

‘Document  HSI-related  system 
performance  issues  (i.e., 

PTRs). 

Production 

and 

Deployment 

All  Domains 
(Integrated) 

Cognitive  Workload, 

Complexity,  Manning, 

Migration  complexity  (Influence 

E. 

Assist  in  T&E  analyses  of 
system  design  and  installation 
(including  LRIP  deficiencies), 

‘Evaluate  system  components 
having  an  impact  on  human 
performance  and  safety. 

Production 

and 

Deployment 

Human 

Factors 

Engineering 

Complexity,  Level  of  service 
requirements  (Subjective 
difficulty  of  satisfying  the  key 

F. 

T&E  support  and  participation 
(lOT&E). 

‘Develop  user  assessment  for 
lOT&E 

“Develop  user-centric 

Production 

and 

Deployment 

All  Domains 
(Integrated) 

Complexity,  Level  of  service 
requirements  (Subjective 
difficulty  of  satisfying  the  key 

G. 

T&E  support  and  participation 
(LFT&E) 

(separated  from  previous 

‘Define  and  test  appropriate 
metrics  (identified  in  JCIDS 
(CDD  and  CPD),  RFP,  etc). 

Production 

and 

Deployment 

All  Domains 
(Integrated) 

Manning:  Testing  Type- 
Campaign  or  force-level. 
Documentation  to  match  Life 

H. 

T&E  support  and  participation 
(FOT&E) 

(separated  from  previous 

‘Develop  user-centric 
assessment  to  support 

FOT&E. 

Production 

and 

Deployment 

All  Domains 
(Integrated) 

Cognitive  Workload, 

Complexity,  Manning, 

Migration  complexity  (Influence 

Figure  40.  Production  and  Development  Phase  HSI  Activities 
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Event 

HSLActivity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

Cost_Drivers 

1. 

Review  OPTEVFOR  reports  and  discuss 
HSi  trade-offs. 

‘Participate  in  an  IPT  and  coordinate  with 
stakehoider. 

‘Verify  and  vaiidate  finai  HFE  reiated 
production  items. 

Production  and 
Deployment 

Human  Factors 
Engineering 

Manning,  Testing  type.  Requirements 
Understanding,  Documentation  to  match 
life  cycle  needs.  Multisite  coordination 

J. 

Document  and  anaiyze  post  depioyment 
iessons  iearned. 

‘Coiiect  and  anaiyze  user  feedback  and 
service  use  data  from  post-depioyment 
system. 

“Anaiyses  of  LRIP  system  performance 

Production  and 
Deployment 

Human  Factors 
Engineering 

Manning,  Documentation  to  match  life 
cycle  needs.  Requirements  understanding. 
Level  of  service  requirements.  Testing, 

Time  constraint  for  System  Development 

K. 

Provide  input  into  review  process  for  P&D 
Phase.  (OTRR,AOTR,PCA;  ‘overview  and 
invoived  participation) 

‘Provide  input  for  assurance  of  operationai 
capabiiities  and  effectiveness  (may  deiay 
FRP)  prior  to  iOT&E  (OTRR)At 
demonstration  (AOTR). 

Production  and 
Deployment 

Human  Factors 
Engineering, 
Survivability, 
Training 

Complexity,  Manning,  Mission,  Operational 
Capability,  Personnel/team  capability 
(Subjective  assessment  of  the  team’s 
intellectual  capability).  Platforms;  Testing 

L. 

Update  Manpower/Personnei  requirements 
for  system  design  and  LCMP. 

‘Perform  anaiysis  and  reporting  on 
Manpower  reiated  test  events,  data 
coiiected,  and  findings. 

‘Revise  the  MER  to  reflect  system  design. 

Production  and 
Deployment 

Manpower, 

Personnel 

Complexity,  Manning,  Mission,  Platforms, 
Specialty  Personnel 

Initial  Training  (Instructor  and  Key 
Personnel),  Mission,  Testing  Type- 

M. 

Update  Training  systems  requirements  for 
system  design  and  LCMP. 

‘Assess  training  solutions  during  initial 
system  component  demonstrations. 

‘Refine  training  solutions  based  on  findings 
from  initial  system  component 

Production  and 
Deployment 

Manpower, 

Personnel, 

Training 

Initial  Training  (Instructor  &  Key 

Personnel),  Initial  Training  (New  Equipment 
at  system  fielding).  Manning,  Mission, 
Occupational  Training,  Training  Analysis 

N. 

Assist  with  System  Hazard  Anaiysis 
(SHA)and  Operating  and  Support  Hazard 
Anaiysis  (O&SHA)  and  System  Safety 
Anaiysis  (SSA). 

‘Participate  in  system  test  events. 

“Perform  analyses  and  reporting  on  Safety 
and  Occupational  Health  (Health  Hazards) 
related  test  events,  data  collected,  and 

Production  and 
Deployment 

HFE,  Habitability, 
ESOH  (Health 
Hazards,  Systems 
Safety) 

Manning,  Testing  Type-system  simulation 
and  stimulation.  Specialty  Personnel 

Figure  41 .  Production  and  Development  Phase  HSI  Activities  (continued) 
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HSI_Activity 

HSI_SubActivity 

Acquisition 
Management  Phase 

Domain 

Cost  Drivers 

A. 

HSI  Planning  and 
Management 

*Provide  updates  to  the  HSI 
Plan. 

*Provide  Status  reports  and 

Operation  & 
Support 

All  Domains 

(Integrated 

Assessment) 

Manning,  Requirements 
understanding. 
Documentation  to  match  life 

B. 

Provide  follow-on  Test  and 
Evaluation  support. 

*Collect  and  analyze  user 
feedback  and  service  use 
data  from  the  post-deployed 

Operation  & 
Support 

Human  Factor 
Engineering 

Cognitive  Workload, 
Complexity,  Criticality  of 
Task,  Difficulty  of  Task, 

C. 

Provide  analysis  for  HFE 
FOC  support  and 
logistics/sustainment 

*Provide  HFE  based 
recommendations  for 
system  modernization  and 

Operation  & 
Support 

Human  Factor 

Engineering 

Habitibility 

Cognitive  Workload, 
Complexity,  Criticality  of 
Task,  Difficulty  of  Task, 

D. 

Provide  recommendations 
for  HFE  FOC  support  and 
logistics/sustainment 

*Provide  HFE  based 
recommendations  for 
system  modernization  and 

Operation  & 
Support 

All  Domains 

(Integrated 

Assessment) 

Complexity,  Maintenance 
concept.  Maintenance 
Requirement, 

E. 

Provide  Training  FOC 
support  and 
logistics/sustainment 

*Transition  post-fielding 
Training  Evaluation 

Analysis  (PFTEA). 

Operation  & 
Support 

Training 

Complexity,  Manning, 
Modeling  &  Simulation, 
Difficulty  of  Task,  Criticality 

F. 

Provide 

Manpower/Personnel  FOC 
support  and 

*Verify  measure  of 
effectiveness  survey. 

*Verify  system  installations 

Operation  & 
Support 

Manpower, 

Personnel 

Manning  (Maintenance 
concept.  Requirements 
understanding.  Migration 

G. 

Analyze  and  prototype  post¬ 
product  improvements  for 
the  next  inremental  build. 

*Assess  Human-Machine 
interfaces,  Human- 
Computer  Interfaces, 

Operation  & 
Support 

Human  Factor 
Engineering 

Cognitive  Workload, 
Complexity,  Criticality  of 
Task,  Difficulty  of  Task, 

H. 

Provide  input  for 
sustainment  of  operational 
readiness  and  safety  of 

*Provide  modeling  and 
simulation  process  or 
improvement  during  FRP 

Operation  & 
Support 

Human  Factor 

Engineering, 

Manpower, 

Manning, 

Model  ing&Simulation, 
Testing  Type-campaign  or 

I. 

Provide  testing  and 
verification  of  follow-on  T&E 
processes. 

Verify  suitability  and 
effectiveness  for  system. 
*(lnclude  availability. 

Operation  & 
Support 

Human  Factor 

Engineering, 

Manpower, 

Manning;  Maintenance 
concept.  Requirements 
understanding,  Physial 

J. 

Provide  FOC  support  and 
logistics/sustainment 
transistion  and  Operation 

*Provide  updates  for  ESOH 
strategy  and  PESHE  in 
conjuction  with  SE 

Operation  & 
Support 

Environment, 
Occupational 
Health,  System 

Manning;  Requirements 
understanding.  Speciality 
Personnel,  Maintenance 

K. 

Provide  input  into  O&S 
review  processes  (ISR) 

*Provide  measurable 
assessments  for  risk, 
readiness,  technical  status 

Operation  & 
Support 

Human  Factor 

Engineering, 

Manpower, 

Manning  (Modeling  & 
Simulation,  Mission, 
Migration  Complexity, 

Figure  42.  Operation  and  Support  HSI  Activities 
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APPENDIX  D.  REVIEW  INSTRUCTIONS 


(Note:  For  THESIS-VIEW  ONLY  of  Review  Instructions.  For  examples,  refer  to  Figures  42 

and  43,  and  Tables  27  and  28) 


Human  System  Integration  Activity  SME  Review 

Background: 

Research  is  underway  at  the  Naval  Postgraduate  School  to  facilitate  Human  System 
Integration  (HSI)  cost  estimation  (CE)  in  the  Analysis  of  Alternative  (AoA)  portion  of  the  Defense 
Acquisition  process.  The  goal  is  to  develop  a  database  of  HSI  CE  activities  that  spans  the  entire 
Acquisition  life  cycle  and  supports  the  cost  estimation  processes. 

You  have  been  identified  as  a  Subject  Matter  Expert  (SME)  in  HSI,  an  HSI  domain  or 
cost  estimation.  We  request  you  review  a  selected  (assigned)  portion  of  the  database  and  provide 
us  feedback.  Any  questions  related  to  this  request  can  be  directed  to  LT  Deborah  Sindall,  USN  at 
dasindal@nps.edu  and  by  telephone  571.830.4971. 

Instructions  to  the  Reviewer:  (2  Parts) 

1 .  Column  Headings  for  Access  Database. 

Review  the  columns  that  are  proposed  for  the  Access  database.  Refer  to  Excel  file  Tab  1 
(Column  Descriptions)  or  Table  1  on  Page  4  of  these  instructions  for  a  list  of  columns  and 
descriptions. 

a.  On  Excel  file  Tab  1  (Column  C),  provide  comments  regarding  additions,  deletion  or 
modification  of  columns  for  the  database.  An  example  is  provided  on  Page  3  of 
these  instructions.  Note:  Data  are  not  system  specific. 

b.  If  a  column  should  be  added,  add  the  item  to  the  bottom  of  the  list. 

2.  HSI  Activities  in  Defense  Acquisition  Phases. 

For  selected  rows  and  columns  on  Excel  file  (Tab  2,  HSI  Activity),  review  the  content  to 
ensure  it  is  appropriate  for  cost  estimation  of  HSI  activities  in  AoA.  A  blank  row  for  each 
activity  has  been  provided  for  your  comments.  If  no  correction  is  needed,  please  indicate 
this  in  the  row  provided.  Examples  of  comments  and  corrections  have  been  provided  on 
Page  3.  Note:  You  will  be  given  one  or  two  phase  activities  on  the  Excel  file  for 
review. 

a.  For  Column  A  (HSI  Activity)  and  B  (HSI  Sub  Activity): 

i.  For  each  row,  determine  if  this  is  a  major  HSI  activity.  If  so,  is  the  activity 
worded  properly?  If  incorrect,  provide  a  recommendation  in  the  comment 
row. 
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ii.  Determine  if  the  Sub  Activity  is  appropriate  for  the  Major  HSI  activity.  If 
incorrect,  recommend  additions,  deletions  or  modifications  in  the  comment 
row. 

b.  For  Column  C,  Acquisition  Phase: 

i.  Determine  whether  the  acquisition  phase  is  correct  for  the  activity.  If  activity 
occurs  during  another  phase,  indicate  appropriate  phase  in  comment  row 
below  activity. 

c.  For  Column  D  (HSI  Domain  {Impacted)): 

i.  Determine  if  listed  domains  are  affected  by  HSI  activity  and  sub  activity.  If 
incorrect,  recommend  additions  or  deletions  of  domains  in  the  comment  row. 

d.  For  Column  E  (%  Effort): 

i.  On  a  100%  scale,  estimate  to  the  best  of  our  ability,  what  percentage  of  man 
hours  is  spent  on  this  activity  for  a  generic  ACAT  I  system.  Your  percentages 
should  add  to  100%  for  each  phase.  (We  realize  that  HSI  efforts  will  vary 
widely  depending  on  the  type  of  system.  This  column  represents  an  initial 
attempt  at  determining  level  of  effort  required  for  HSI  practitioners.) 

e.  For  Column  F  (Cost  Drivers): 

i.  Review  the  list  of  Cost  Drivers  on  Excel  file  Tab  3  (Cost  Drivers).  Also 
available  on  Page  5  of  these  instructions. 

ii.  Determine  if  listed  potential  cost  drivers  are  appropriate  for  activity  listed.  If 
drivers  listed  are  incorrect,  recommend  additions  or  deletions  for  the  activity 
in  the  comment  row.  If  a  cost  driver  is  not  listed  but  should  be  included, 
please  provide  the  driver  in  comment  row. 

f.  For  Missing  Activities: 

i.  If  an  HSI  activity  or  sub-activity  has  been  left  out  of  the  database,  please  add 
it  to  the  bottom  of  the  activity  list.  Fill  out  as  much  information  for  the 
columns  as  possible  to  ensure  the  activity  it  is  represented  appropriately. 


3.  Save  the  document  using  Excel  (97-2003)  compatible  file  type  using 
“samefilename_yourinitials.xls”  format.  Please  send  to  dasindal@nps.edu  no  later  than  31 
Aug  2010.  If  more  time  is  needed,  please  e-mail  with  request. 

Sources  of  HSI  Data: 

HSI  CE  activity  development  was  determined  through  analysis  of  current  instruction,  guidance, 
integrated  frameworks  and  past  HSI  cost  studies.  Please  see  Page  6  of  these  instructions  for  a 
list  of  resources. 
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WBS  Cost  Element 

Numeric  Code  for  organization  of  Activities  based 
on  Specific  Program  Work  Breakdown  Cost 
Structure  (Procurement,  O&S,  O&M,  RDT&E, 
Spares) 

Example  Comment:  WBS  stmcture  is  generic 
and  will  be  taylored  down  for  specific  systems. 

Acquisition  Phase 

6  Phases  total:  Pre-Material  Solution  Analysis 
(Pre-MSA);  Material  Solution  Analysis  (MSA), 
Technology  Development  Phase  (TD); 

Engineering  &  Manufacturing  Development 

Phase:  Integrated  System  Design,  System 
Capability  and  Manufacturing  Process 
Demonstration  (EMD);  Production  &  Deployment 
(PD);  Operation  and  Support  (O&S) 

Figure  43. 


Column  Heading  Description  Example  -Q1  (From  Review  Instructions  Page  4) 
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HSI  Actrvity-SME  REVIEW  -  Microsoft  Excel 
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HSI_Activity 

HSI_SubActivity 

Acquisition 

Management 

Phase 

Domain 

%  Effort 

(Man  hours) 

Cost_Orivers 

- 

DATA 

Establish  a  HSI  IPT 

Participate  in  IPT|  Incorporate  Users,  SMEs  Engineers,  Contractors  / 
Government  HSI  representatives  and  other  HSI  domain  experts  into 
Integrated  Performance  Teams). 

Support  User-Centric  Design  Process 

Technology 

Development 

Environment,  Habitability, 
Human  Factor  Engineering, 
Manpower,  Occupational 
Health,  Personnel, 
Survivability,  System  Safety, 
Training 

Manning,  Personnel  experience/continuity 
(Subjective  assessmentof  staff  consistency). 
Requirements  understanding  (Subjective 
assessment  of  the  system  requirements). 

Target  Audience 

pynmplR 

Input 

This  maybcpartofa  bgger  activity. 

Need  to  exapnd  "Support  User-Centric  Design  Process 

Ex.  Training  rtot  involved, 

omit.  Ex 

5K  man  hours 

required  of  lOOW 

Manning’  can  be  used  for  actual  work 
completed  (Man  hours  to  perform  in  IPT)  or  as  a 
driver  for  system  action  to  be  performed  on 

Example 

input 

All  column  inputs  are  acceptable. 

Figure  44.  HSI  Activity  Review  Example  -Q2  (From  Review  Instructions  Page  3) 
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Table  27.  SME  Review  Potential  Column  Names  and  Descriptions  for  HSI  Activity 
Database  Development  (From  Review  Instructions,  Page  4) 


Column  Name 

Descriptbn 

WBS  Cost 
Element 

Numeric  Code  for  organization  of  Activities  based  on  Specific  Program  Work  Breakdown  Cost 
Structure  (Procurement,  O&S,  O&M,  RDT&E,  Spares) 

Acquisition 

Phase 

7  Phases  total:  Pre-Material  Solution  Analysis;  Material  Solution  Analysis,  Technology 
Development  Phase;  Engineering  &  Manufacturing  Development  Phase;  Integrated  System 
Design,  System  Capability  and  Manufacturing  Process  Demonstration;  Production  &  Deployment; 
Operation  and  Support 

Event  Category 

Organization  of  Phase  activities  based  on  IDTA&L  Defense  Interactive  Framework  (JCIDS  Joint 
Capabilities  Integration  and  Development  System,  Information  Technology  (IT)  &  National  Security 
(NSS),  Earned  Value  Management  (EVM),  Oversights  Review,  Contracting,  Major  Products,  Life 
Cycle  Logistics/Sustainment,  Technical,  Cost 

HSI  Activity 

HSI  practitioners  expected  contribution  to  acquisition  process. 

HSI  Sub 

Activity 

Detail  level  orsubset  of  HSI  Activity 

%  Effort 

On  a  scale  of  1-100%  for  each  ph  ase,  the  level  of  HSI  practitioner  effort  contributed  in  program 
manageme  nt. 

For  Example:  ActivitvOne"  Provide  HSI  activity  inputdurinq  Capabilities  Based  Assessment  for 

ICD  and  CDD”,  HSI  activity  is  only  3%  of  man-hour  effort  for  WEAPONS  (TRACKED  VEH) 
PROCUREMENT  (2033  -  WTCV). 

HSI  Domain 
(Impacted) 

HSI  domain  affected  by  the  activity  being  performed,  (practitioner,  IPT,  or  domain  involvement) 
Environment,  Habitability,  Human  Factor  Engineering,  Manpower,  Occupational  Health,  Personnel, 
Survivability,  System  Safety,  Training. 

System  T ype 

T ype  of  system  /program  that  HSI  activities  will  be  performed  on  (i.e  .  aviation  ,  ship  building, 
weapon  system ,  etc). 

Critical  Issues 

Operation  Effectiveness  or  Technical  Issues  that  will  affect  Test  and  Evaluation  Processes  (i.e. 
Survivability,  Reliability) 

Measures  of 
Effectiveness 

Measure  designed  to  correspond  to  accomplishment  of  mission  objectives  and  achievement  of 
desired  results. 

Measure  of 
Performance  / 
Suitability 

MOP:  Measure  of  a  system's  performance  expressed  as  speed,  payload,  range,  time  on  station, 
frequency,  or  other  distinctly  quantifiable  performance  features.  MOS:  Measure  of  an  item's  ability 
to  be  supported  in  its  intended  operational  environment.  MOSs  typically  relates  to  readiness  or 
operational  availability,  and  hence  neliabiity,  maintainability,  and  the  item's  support  structure. 

Cost  Drivers 

An  element  that  impacts  the  cost  of  an  activity  (i.e.  Manning,  Cognitive  Workload,  Specialty 
Personnel)  '‘Will  be  used  in  combination  with  MOE/MOP  to  develop  potential  Cost  Element  and 

Cost  Element  Relatbnships  for  Database  Development* 

Benefit 

It  defines  a  solution  aimed  at  achieving  specific  organizational  objectives  by  quantifying  the 
potential  financial  impacts  and  other  business  benefits  such  as:  Savings  and/or  cost  avoidance. 
Revenue  enhancements  and/or  cash-flow  improvements.  Performance  improvements.  Are  listed 
as  quantifiable/non-quantifiable.  Used  to  develop  Cost  Element  Relationships. 

Cost 

Estimation 

Type 

'Program  Office  Estimate.  'Component  Cost  Estimate.  'Independent  Cost  Estimate:  *  Life-cycle 
cost  estimate.  'Total  Ownership  CE:  estimate  consists  of  the  elements  of  life-cycle  cost  plus  some 
infrastructure  and  business  process  costs  not  necessarily  attributable  to  a  program.  'Rough  Order 
of  Magnitu  de.  'Independent  Cost  Assessment,  Independent  Govern  ment  Cost  Estimate. 

Cost 

Estimation 

Relationships 

complex  equations  with  a  number  of  independent  variables.  Or  Cost  Factors  (single  independent 
variable).  Ratbs  or  percentages  use  in  cost  estimation  (i.e.  parametericcost  estimatbn  methods). 

Im  pact 

Potential  Impact  of  benefit  provided.  Used  to  determine  Trade-off.  Database  will  include  Benefit 
Library  with  Benefit  ID,  Description,  and  Category  (guantifiable,  non-guantifiable) 

Trade-off 

Potentially  weighted  domains  or  benefits  that  are  used  for  cost  over  another.  Baed  on 
prog  ram /system 

Comments 

Process,  Detail  or  integration  notes  relevant  to  HSI  Activity  or  Sub  Activity. 

Target 

Documents 

Source  or  projected  document  for  which  HSI  activity  occurs  (Ex:  Contractor  Cost  Data  Reports 

Cost  Performance  Reports,  Selected  Acquisition  Reports,  Recent  Defense  Acquisition  Executive 
Summaries,  Unit  Cost  Reports,  Manpower  Estimate  Reports,  HSIP,  TEMP,  SEP,  ITR  etc. 

References 

Resources  for  further  database  user  information. 
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Table  28.  SME  Review  Potential  Cost  Drivers  (from  Review  Instructions,  Page  5) 


Cbgi  itive  Workload 

Cbmplexity 

QilicalityofTask 

□ffcUtyofTask 

Cbcumertation  to  match  ffe  cyde  needs  (Bneadthand  depthofrequied  doc umentstion) 

Frequency  of  Ta  sk 

hiitial Training  (Instructor  &  Key P erscn ne  1) 

hiitia  1 T rain  ing  (N  ew  Equ  i  pm  en  t  a  t  syste  m  f  iel  d  ng) 

Level  of  service  requirements  (SulqectivedifficulV  of  satisfying  the  key  performance  Parameters) 

IVfointenance  Concept 

Nteintenanoe  Ftequirerrent 

IVfonnhg 

Mgafon  ccmplexity(lrfluenceof  legacysistem  (if  applicabfe)) 

Mssicn 

Mcdelng&  Simulatbn 

Mjitisitecocrdination  (Locationof  stakeholders  andcocrdndion  harriers) 

Occupatbnal  Trading 

Operational  Cepebility 

Ftersonnel  s<perience/oortinLity(Subjective  assessment  of  staff  consistency) 

Ftersonn  el/team  capebility  (SubjecSve assessment  of  the  team’sirteltecfoal  capabilty) 

Physical  Workload 

Reforms  (ships,  aircraft,  weapon  systems,  etc) 

Fhocess  capability  (CMMI  level  or  equivdent  rding) 

Fte  qu  ire  rre  nts  untte  islanding  (Subpcfve  assessment  of  the  system  req  Jrements) 

Sze 

SLOG 

SpecidtyP  erscn  nel 

StakehokJerteam  cohesion  (Subjective  assessment  ofa II stekeho tiers) 

Target  Audie  noe 

Technology  risk  (Nteturity,  readiness,  an  d  obsole  seen  ce  of  technobg  y) 

Time  Constraint  fa  System  Development 

Tod  support  (Sut^'ective  assessmentof  SE  tods) 

lasting  lype-detaled  component  testing 

Testing  Type-subsystem  integrdion 

Testing  Type-system  simdationard  stimuldion 

Testing  Type-campaian  or  force-level  (system  ofsvsterrBi. 

Type 

Lh it/Susla inm ent  FiddTranhg 

Warranties 
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